# MaxUp: A Simple Way to Improve Generalization of Neural Network Training

arXiv: https://arxiv.org/abs/2002.09024 · HTML: https://mlsanity.com/paper/2002.09024/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Feb 2020.

## Built on by (most cited first, 27 of 27)

- [High-Performance Large-Scale Image Recognition Without Normalization](https://mlsanity.com/paper/2102.06171/) (2021)
- [AugMax: Adversarial Composition of Random Augmentations for Robust Training](https://mlsanity.com/paper/2110.13771/) (2021)
- [Data-free Backdoor Removal based on Channel Lipschitzness](https://mlsanity.com/paper/2208.03111/) (2022)
- [Vision Transformers with Patch Diversification](https://mlsanity.com/paper/2104.12753/) (2021)
- [DP-InstaHide: Provably Defusing Poisoning and Backdoor Attacks with Differentially Private Data Augmentations](https://mlsanity.com/paper/2103.02079/) (2021)
- [Noisy Feature Mixup](https://mlsanity.com/paper/2110.02180/) (2021)
- [Taxonomy of Machine Learning Safety: A Survey and Primer](https://mlsanity.com/paper/2106.04823/) (2021)
- [Indiscriminate Data Poisoning Attacks on Neural Networks](https://mlsanity.com/paper/2204.09092/) (2022)
- [Removing Batch Normalization Boosts Adversarial Training](https://mlsanity.com/paper/2207.01156/) (2022)
- [ALLSH: Active Learning Guided by Local Sensitivity and Hardness](https://mlsanity.com/paper/2205.04980/) (2022)
- [What Doesn't Kill You Makes You Robust(er): How to Adversarially Train against Data Poisoning](https://mlsanity.com/paper/2102.13624/) (2021)
- [On the Inherent Regularization Effects of Noise Injection During Training](https://mlsanity.com/paper/2102.07379/) (2021)
- [Run-Off Election: Improved Provable Defense against Data Poisoning Attacks](https://mlsanity.com/paper/2302.02300/) (2023)
- [Distilling Out-of-Distribution Robustness from Vision-Language Foundation Models](https://mlsanity.com/paper/2311.01441/) (2023)
- [Accelerating Self-Supervised Learning via Efficient Training Strategies](https://mlsanity.com/paper/2212.05611/) (2022)
- [GAT: Guided Adversarial Training with Pareto-optimal Auxiliary Tasks](https://mlsanity.com/paper/2302.02907/) (2023)
- [Ortho-Shot: Low Displacement Rank Regularization with Data Augmentation for Few-Shot Learning](https://mlsanity.com/paper/2110.09374/) (2021)
- [Boosting Factorization Machines via Saliency-Guided Mixup](https://mlsanity.com/paper/2206.08661/) (2022)
- [Improving Robustness of Learning-based Autonomous Steering Using Adversarial Images](https://mlsanity.com/paper/2102.13262/) (2021)
- [Generalising via Meta-Examples for Continual Learning in the Wild](https://mlsanity.com/paper/2101.12081/) (2021)
- [SanitAIs: Unsupervised Data Augmentation to Sanitize Trojaned Neural Networks](https://mlsanity.com/paper/2109.04566/) (2021)
- [HCDG: A Hierarchical Consistency Framework for Domain Generalization on Medical Image Segmentation](https://mlsanity.com/paper/2109.05742/) (2021)
- [Preventing Model Collapse in Deep Canonical Correlation Analysis by Noise Regularization](https://mlsanity.com/paper/2411.00383/) (2024)
- [On Hard Episodes in Meta-Learning](https://mlsanity.com/paper/2110.11190/) (2021)
- [Multi-Augmentation for Efficient Visual Representation Learning for Self-supervised Pre-training](https://mlsanity.com/paper/2205.11772/) (2022)
- [Improving Generalization in Meta-Learning via Meta-Gradient Augmentation](https://mlsanity.com/paper/2306.08460/) (2023)
- [Hazards in Deep Learning Testing: Prevalence, Impact and Recommendations](https://mlsanity.com/paper/2309.05381/) (2023)

## Abstract (arXiv)

We propose MaxUp, an embarrassingly simple, highly effective technique for improving the generalization performance of machine learning models, especially deep neural networks. The idea is to generate a set of augmented data with some random perturbations or transforms and minimize the maximum, or worst case loss over the augmented data. By doing so, we implicitly introduce a smoothness or robustness regularization against the random perturbations, and hence improve the generation performance. For example, in the case of Gaussian perturbation, MaxUp is asymptotically equivalent to using the gradient norm of the loss as a penalty to encourage smoothness. We test MaxUp on a range of tasks, including image classification, language modeling, and adversarial certification, on which MaxUp consistently outperforms the existing best baseline methods, without introducing substantial computational overhead. In particular, we improve ImageNet classification from the state-of-the-art top-1 accuracy 85.5% without extra data to 85.8%. Code will be released soon.