mlsanityData to 7 Oct 2026

Implicit Gradient Regularization

arXiv 2009.11162 ↗First posted 23 Sep 2020cs.LG stat.MLcited by 145

Not on the map. Why?

Abstract (arXiv)

Gradient descent can be surprisingly good at optimizing deep neural networks without overfitting and without explicit regularization. We find that the discrete steps of gradient descent implicitly regularize models by penalizing gradient descent trajectories that have large loss gradients. We call this Implicit Gradient Regularization (IGR) and we use backward error analysis to calculate the size of this regularization. We confirm empirically that implicit gradient regularization biases gradient descent toward flat minima, where test errors are small and solutions are robust to noisy parameter perturbations. Furthermore, we demonstrate that the implicit gradient regularization term can be used as an explicit regularizer, allowing us to control this gradient regularization directly. More broadly, our work indicates that backward error analysis is a useful theoretical approach to the perennial question of how learning rate, model size, and parameter regularization interact to determine the properties of overparameterized models optimized with gradient descent.

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 25; 2023: 38; 2024: 22; 2025: 23; 2026: 20.

145 papers.

Sort:

By area:

  1. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  2. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  3. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  4. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  5. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  6. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  7. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  8. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  9. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  10. 2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
  11. 2024Normalization and effective learning rates in reinforcement learningcited by 55
  12. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
38 more
  1. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 47
  2. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  3. 2021Subspace Adversarial Trainingcited by 41
  4. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 38
  5. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  6. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 38
  7. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  8. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 36
  9. 2023Deep networks for system identification: a Surveycited by 35
  10. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  11. 2023Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferabilitycited by 32
  12. 2023Brain-inspired Computational Intelligence via Predictive Codingcited by 30
  13. 2021Implicit Gradient Alignment in Distributed and Federated Learningcited by 29
  14. 2023Flatness-Aware Minimization for Domain Generalizationcited by 27
  15. 2021Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysiscited by 25
  16. 2023Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descentcited by 25
  17. 2023On the Implicit Bias of Adamcited by 25
  18. 2022An Empirical Study of Implicit Regularization in Deep Offline RLcited by 24
  19. 2022Implicit regularization of dropoutcited by 24
  20. 2022Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensionscited by 23
  21. 2025Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimizationcited by 21
  22. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  23. 2023Sparse Modular Activation for Efficient Sequence Modelingcited by 20
  24. 2021The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusioncited by 19
  25. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
  26. 2024Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimizationcited by 17
  27. 2022Understanding Gradient Regularization in Deep Learning: Efficient Finite-Difference Computation and Implicit Biascited by 16
  28. 2023The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networkscited by 15
  29. 2021Discretization Drift in Two-Player Gamescited by 14
  30. 2023Differentially Private Sharpness-Aware Trainingcited by 14
  31. 2025Training Dynamics Impact Post-Training Quantization Robustnesscited by 13
  32. 2022FINDE: Neural Differential Equations for Finding and Preserving Invariant Quantitiescited by 12
  33. 2022Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysiscited by 12
  34. 2024Neglected Hessian component explains mysteries in Sharpness regularizationcited by 12
  35. 2025The Rich and the Simple: On the Implicit Bias of Adam and SGDcited by 12
  36. 2022Learning Dynamics and Generalization in Reinforcement Learningcited by 11
  37. 2022Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal Statescited by 11
  38. 2023On a continuous time model of gradient descent dynamics and instability in deep learningcited by 11

50 most cited of 145 All 145 (JSON)