mlsanityData to 13 Oct 2026

Understanding Gradient Regularization in Deep Learning: Efficient Finite-Difference Computation and Implicit Bias

arXiv 2210.02720 ↗First posted 6 Oct 2022cs.LG stat.MLcited by 16

Not on the map. Why?

Abstract (arXiv)

Gradient regularization (GR) is a method that penalizes the gradient norm of the training loss during training. While some studies have reported that GR can improve generalization performance, little attention has been paid to it from the algorithmic perspective, that is, the algorithms of GR that efficiently improve the performance. In this study, we first reveal that a specific finite-difference computation, composed of both gradient ascent and descent steps, reduces the computational cost of GR. Next, we show that the finite-difference computation also works better in the sense of generalization performance. We theoretically analyze a solvable model, a diagonal linear network, and clarify that GR has a desirable implicit bias to so-called rich regime and finite-difference computation strengthens this bias. Furthermore, finite-difference GR is closely related to some other algorithms based on iterative ascent and descent steps for exploring flat minima. In particular, we reveal that the flooding method can perform finite-difference GR in an implicit way. Thus, this work broadens our understanding of GR for both practice and theory.

Key facts. Cited by 16 papers in the map (data through 13 Oct 2026). Builds on 17 of its 25 references that are in the map. First posted on arXiv 6 Oct 2022.

Builds on

17 of 25 references.

References · 17

  1. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,411
  2. 2018On Lazy Training in Differentiable Programmingcited by 675
  3. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 305
  4. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  5. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  6. 2020Implicit Gradient Regularizationcited by 146
  7. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  8. 2019Robust Learning with Jacobian Regularizationcited by 127
  9. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  10. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 109
  11. 2021The Efficiency Misnomercited by 98
  12. 2020The Hessian Penalty: A Weak Prior for Unsupervised Disentanglementcited by 90
5 more
  1. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 62
  2. 2020Do We Need Zero Training Loss After Achieving Zero Training Error?cited by 61
  3. 2020Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalizationcited by 61
  4. 2019Scaleable input gradient regularization for adversarial robustnesscited by 45
  5. 2019Information-Theoretic Local Minima Characterization and Regularizationcited by 14

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 2; 2026: 5.

16 papers.

Sort:

By area:

  1. 2023Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferabilitycited by 32
  2. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewardscited by 8
  3. 2026Detecting and Suppressing Reward Hacking with Gradient Fingerprintscited by 5
  4. 2024Exploring Loss Landscapes through the Lens of Spin Glass Theorycited by 4
  5. 2024Preconditioning for Accelerated Gradient Descent Optimization and Regularizationcited by 4
  6. 2025OMP: One-step Meanflow Policy with Directional Alignmentcited by 4
  7. 2023Soft ascent-descent as a stable and flexible alternative to floodingcited by 3
  8. 2023An Infinite-Width Analysis on the Jacobian-Regularised Training of a Neural Networkcited by 3
  9. 2024When Will Gradient Regularization Be Harmful?cited by 3
  10. 2023Bias-Aware Minimisation: Understanding and Mitigating Estimator Bias in Private SGDcited by 1
  11. 2026Contractive Diffusion Policies: Robust Action Diffusion via Contractive Score-Based Sampling with Differential Equationscited by 1
  12. 2026Gradient Regularized Natural Gradientscited by 1
4 more
  1. 2024Criterion Collapse and Loss Distribution Controlcited by 0
  2. 2024Making Robust Generalizers Less Rigid with Loss Concentrationcited by 0
  3. 2025First-ish Order Methods: Hessian-aware Scalings of Gradient Descentcited by 0
  4. 2026Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementationcited by 0