mlsanityData to 7 Oct 2026

Why Do We Need Weight Decay in Modern Deep Learning?

arXiv 2310.04415 ↗First posted 6 Oct 2023cs.LGcited by 70

Not on the map. Why?

Abstract (arXiv)

Weight decay is a broadly used technique for training state-of-the-art deep networks from image classification to large language models. Despite its widespread usage and being extensively studied in the classical literature, its role remains poorly understood for deep learning. In this work, we highlight that the role of weight decay in modern deep learning is different from its regularization effect studied in classical learning theory. For deep networks on vision tasks trained with multipass SGD, we show how weight decay modifies the optimization dynamics enhancing the ever-present implicit regularization of SGD via the loss stabilization mechanism. In contrast, for large language models trained with nearly one-epoch training, we describe how weight decay balances the bias-variance tradeoff in stochastic optimization leading to lower training loss and improved training stability. Overall, we present a unifying perspective from ResNets on vision tasks to LLMs: weight decay is never useful as an explicit regularizer but instead changes the training dynamics in a desirable way. The code is available at https://github.com/tml-epfl/why-weight-decay

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 44 of its 69 references that are in the map. First posted on arXiv 6 Oct 2023.

Builds on

44 of 69 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  12. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
32 more
  1. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  2. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  3. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  4. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  5. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  6. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  7. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  8. 2017Three Factors Influencing Minima in SGDcited by 270
  9. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 227
  10. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  11. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 136
  12. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  13. 2017L2 Regularization versus Batch and Weight Normalizationcited by 121
  14. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  15. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  16. 2018Three Mechanisms of Weight Decay Regularizationcited by 112
  17. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  18. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  19. 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 81
  20. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  21. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  22. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  23. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  24. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 50
  25. 2020On the training dynamics of deep networks with L2 regularizationcited by 38
  26. 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 38
  27. 2021Power-law escape rate of SGDcited by 29
  28. 2022Robust Training of Neural Networks Using Scale Invariant Architecturescited by 29
  29. 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
  30. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
  31. 2020Understanding Decoupled and Early Weight Decaycited by 12
  32. 2021How to decay your learning ratecited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 14; 2025: 30; 2026: 25.

70 papers.

Sort:

By area:

  1. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  2. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  3. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  4. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
  5. 2025Generalized Interpolating Discrete Diffusioncited by 68
  6. 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 42
  7. 2025Grokking at the Edge of Numerical Stabilitycited by 41
  8. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 41
  9. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 34
  10. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  11. 2025Pre-training under infinite computecited by 24
  12. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 21
38 more
  1. 2025Why Gradients Rapidly Increase Near the End of Trainingcited by 21
  2. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  3. 2025Generalized Gradient Norm Clipping & Non-Euclidean (L0,L1)-Smoothnesscited by 17
  4. 2024Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignmentcited by 15
  5. 2024How Much Can We Forget about Data Contamination?cited by 14
  6. 2025AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMscited by 12
  7. 2025Cautious Weight Decaycited by 12
  8. 2026Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimizationcited by 12
  9. 2025A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisationcited by 11
  10. 2024Switch EMA: A Free Lunch for Better Flatness and Sharpnesscited by 9
  11. 2025The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditionscited by 9
  12. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 8
  13. 2025Robust Layerwise Scaling Rules by Proper Weight Decay Tuningcited by 8
  14. 2026A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMscited by 8
  15. 2025FOCUS: First Order Concentrated Updating Schemecited by 7
  16. 2025GAIA: A Global, Multi-modal, Multi-scale Vision-Language Dataset for Remote Sensing Image Analysiscited by 7
  17. 2025OUI Need to Talk About Weight Decay: A New Perspective on Overfitting Detectioncited by 7
  18. 2025Navigating the Latent Space Dynamics of Neural Modelscited by 7
  19. 2024Understanding the Generalization Benefits of Late Learning Rate Decaycited by 6
  20. 2024Where Do Large Learning Rates Lead Us?cited by 6
  21. 2024Clustering and Alignment: Understanding the Training Dynamics in Modular Additioncited by 5
  22. 2023Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescalingcited by 4
  23. 2024Data Diversity as Implicit Regularization: How Does Diversity Shape the Weight Space of Deep Neural Networks?cited by 4
  24. 2025SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Trainingcited by 4
  25. 2025Efficient Hyperparameter Tuning via Trajectory Invariance Principlecited by 4
  26. 2026The Effect of Mini-Batch Noise on the Implicit Bias of Adamcited by 4
  27. 2026Weight Decay Improves Language Model Plasticitycited by 4
  28. 2024The Fair Language Model Paradoxcited by 3
  29. 2025Controlling the Flow: Stability and Convergence for Stochastic Gradient Descent with Decaying Regularizationcited by 3
  30. 2026Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectorscited by 3
  31. 2024Hallmarks of Optimization Trajectories in Neural Networks: Directional Exploration and Redundancycited by 2
  32. 2025How does the optimizer implicitly bias the model merging loss landscape?cited by 2
  33. 2025Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Self-Attention Transformerscited by 2
  34. 2026Learnable Multipliers: Freeing the Scale of Language Model Matrix Layerscited by 2
  35. 2026Quantifying LLM Attention-Head Stability: Implications for Circuit Universalitycited by 2
  36. 2026Disentangling Geometry, Performance, and Training in Language Modelscited by 2
  37. 2026Beyond Linearity in Attention Projections: The Case for Nonlinear Queriescited by 2
  38. 2026Neural Weight Norm = Kolmogorov Complexitycited by 2

50 most cited of 70 All 70 (JSON)