mlsanityData to 14 Oct 2026

Three Mechanisms of Weight Decay Regularization

arXiv 1810.12281 ↗First posted 29 Oct 2018cs.LG stat.MLcited by 113

Not on the map. Why?

Abstract (arXiv)

Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of L2 regularization. Literal weight decay has been shown to outperform L2 regularization for optimizers for which they differ. We empirically investigate weight decay for three optimization algorithms (SGD, Adam, and K-FAC) and a variety of network architectures. We identify three distinct mechanisms by which weight decay exerts a regularization effect, depending on the particular optimization algorithm and architecture: (1) increasing the effective learning rate, (2) approximately regularizing the input-output Jacobian norm, and (3) reducing the effective damping coefficient for second-order optimization. Our results provide insight into how to improve the regularization of neural networks.

Key facts. Cited by 113 papers in the map (data through 14 Oct 2026). First posted on arXiv 29 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 20; 2022: 26; 2023: 23; 2024: 17; 2025: 10; 2026: 17.

113 papers.

Sort:

By area:

  1. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  2. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 402
  3. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 146
  4. 2023Explaining grokking through circuit efficiencycited by 111
  5. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 106
  6. 2022Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networkscited by 95
  7. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  8. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 72
  9. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
  10. 2021SimPLE: Similar Pseudo Label Exploitation for Semi-Supervised Classificationcited by 68
  11. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  12. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
38 more
  1. 2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 58
  2. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
  3. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 53
  4. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 50
  5. 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 43
  6. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 43
  7. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 31
  8. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 30
  9. 2022Robust Training of Neural Networks Using Scale Invariant Architecturescited by 29
  10. 2024Weight decay induces low-rank attention layerscited by 27
  11. 2024Prompt Learning via Meta-Regularizationcited by 26
  12. 2023Grokking in Linear Estimators -- A Solvable Model that Groks without Understandingcited by 25
  13. 2021Tensor Normal Training for Deep Learning Modelscited by 24
  14. 2023MoMo: Momentum Models for Adaptive Learning Ratescited by 24
  15. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 23
  16. 2025Why Gradients Rapidly Increase Near the End of Trainingcited by 22
  17. 2021On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decaycited by 20
  18. 2023Revisiting the Fragility of Influence Functionscited by 20
  19. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  20. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
  21. 2026Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimizationcited by 16
  22. 2023A Stochastic Proximal Polyak Step Sizecited by 15
  23. 2023On the Parameterization of Second-Order Optimization Effective Towards the Infinite Widthcited by 15
  24. 2022A Data-Augmentation Is Worth A Thousand Samples: Exact Quantification From Analytical Augmented Sample Momentscited by 14
  25. 2024How Much Can We Forget about Data Contamination?cited by 14
  26. 2025Provable Separations between Memorization and Generalization in Diffusion Modelscited by 14
  27. 2022Improving Robustness with Adaptive Weight Decaycited by 13
  28. 2022Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysiscited by 13
  29. 2021Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependencecited by 12
  30. 2022Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularizationcited by 12
  31. 2022Semi-Discrete Normalizing Flows through Differentiable Tessellationcited by 11
  32. 2021Edge of chaos as a guiding principle for modern neural network trainingcited by 10
  33. 2023Stochastic Marginal Likelihood Gradients using Neural Tangent Kernelscited by 10
  34. 2023HyperSparse Neural Networks: Shifting Exploration to Exploitation through Adaptive Regularizationcited by 9
  35. 2024On the Nonlinearity of Layer Normalizationcited by 9
  36. 2022A Mini-Block Fisher Method for Deep Neural Networkscited by 8
  37. 2022Cyclical Focal Losscited by 8
  38. 2022Scale-invariant Bayesian Neural Networks with Connectivity Tangent Kernelcited by 7

50 most cited of 113 All 113 (JSON)