mlsanityData to 14 Oct 2026

When Will Gradient Regularization Be Harmful?

arXiv 2406.09723 ↗First posted 14 Jun 2024cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Gradient regularization (GR), which aims to penalize the gradient norm atop the loss function, has shown promising results in training modern over-parameterized deep neural networks. However, can we trust this powerful technique? This paper reveals that GR can cause performance degeneration in adaptive optimization scenarios, particularly with learning rate warmup. Our empirical and theoretical analyses suggest this is due to GR inducing instability and divergence in gradient statistics of adaptive optimizers at the initial training stage. Inspired by the warmup heuristic, we propose three GR warmup strategies, each relaxing the regularization effect to a certain extent during the warmup course to ensure the accurate and stable accumulation of gradients. With experiments on Vision Transformer family, we confirm the three GR warmup strategies can effectively circumvent these issues, thereby largely improving the model performance. Meanwhile, we note that scalable models tend to rely more on the GR warmup, where the performance can be improved by up to 3% on Cifar10 compared to baseline GR. Code is available at hrefhttps://github.com/zhaoyang-0204/gnphttps://github.com/zhaoyang-0204/gnp.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 19 of its 27 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

19 of 27 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2016Layer Normalizationcited by 5,669
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  9. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  10. 2021Going deeper with Image Transformerscited by 764
  11. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  12. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
7 more
  1. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  2. 2020Implicit Gradient Regularizationcited by 146
  3. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  4. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  5. 2018Training Tips for the Transformer Modelcited by 85
  6. 2022Understanding Gradient Regularization in Deep Learning: Efficient Finite-Difference Computation and Implicit Biascited by 16
  7. 2022Randomized Sharpness-Aware Training for Boosting Computational Efficiency in Deep Learningcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 79
  2. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewardscited by 8
  3. 2025Insights from Gradient Dynamics: Gradient Autoscaled Normalizationcited by 0