# GradAug: A New Regularization Method for Deep Neural Networks

arXiv: https://arxiv.org/abs/2006.07989 · HTML: https://mlsanity.com/paper/2006.07989/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2020.

## Built on by (most cited first, 18 of 18)

- [Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2111.14213/) (2021)
- [Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks](https://mlsanity.com/paper/2201.03299/) (2022)
- [A Systematic Review of Robustness in Deep Learning for Computer Vision: Mind the gap?](https://mlsanity.com/paper/2112.00639/) (2021)
- [MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks](https://mlsanity.com/paper/2103.06132/) (2021)
- [Consistency-based Active Learning for Object Detection](https://mlsanity.com/paper/2103.10374/) (2021)
- [Knowledge Distillation for Federated Learning: a Practical Guide](https://mlsanity.com/paper/2211.04742/) (2022)
- [The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillation](https://mlsanity.com/paper/2301.08968/) (2023)
- [Network Augmentation for Tiny Deep Learning](https://mlsanity.com/paper/2110.08890/) (2021)
- [RoPGen: Towards Robust Code Authorship Attribution via Automatic Coding Style Transformation](https://mlsanity.com/paper/2202.06043/) (2022)
- [FedUV: Uniformity and Variance for Heterogeneous Federated Learning](https://mlsanity.com/paper/2402.18372/) (2024)
- [Stimulative Training++: Go Beyond The Performance Limits of Residual Networks](https://mlsanity.com/paper/2305.02507/) (2023)
- [Masking meets Supervision: A Strong Learning Alliance](https://mlsanity.com/paper/2306.11339/) (2023)
- [Boosting Residual Networks with Group Knowledge](https://mlsanity.com/paper/2308.13772/) (2023)
- [Cooperative Meta-Learning with Gradient Augmentation](https://mlsanity.com/paper/2406.04639/) (2024)
- [Selective Output Smoothing Regularization: Regularize Neural Networks by Softening Output Distributions](https://mlsanity.com/paper/2103.15383/) (2021)
- [Improving Generalization in Meta-Learning via Meta-Gradient Augmentation](https://mlsanity.com/paper/2306.08460/) (2023)
- [UniVarFL: Uniformity and Variance Regularized Federated Learning for Heterogeneous Data](https://mlsanity.com/paper/2506.08167/) (2025)
- [Multi-Level Analyzation of Imbalance to Resolve Non-IID-Ness in Federated Learning](https://mlsanity.com/paper/2606.10250/) (2026)

## Abstract (arXiv)

We propose a new regularization method to alleviate over-fitting in deep neural networks. The key idea is utilizing randomly transformed training samples to regularize a set of sub-networks, which are originated by sampling the width of the original network, in the training process. As such, the proposed method introduces self-guided disturbances to the raw gradients of the network and therefore is termed as Gradient Augmentation (GradAug). We demonstrate that GradAug can help the network learn well-generalized and more diverse representations. Moreover, it is easy to implement and can be applied to various structures and applications. GradAug improves ResNet-50 to 78.79% on ImageNet classification, which is a new state-of-the-art accuracy. By combining with CutMix, it further boosts the performance to 79.67%, which outperforms an ensemble of advanced training tricks. The generalization ability is evaluated on COCO object detection and instance segmentation where GradAug significantly surpasses other state-of-the-art methods. GradAug is also robust to image distortions and FGSM adversarial attacks and is highly effective in low data regimes. Code is available at https://github.com/taoyang1122/GradAug