Does Weight Decay Enhance Training Stability?
Not on the map. Why?
Abstract (arXiv)
In modern deep learning, weight decay is often credited with "stabilizing" training dynamics, diverging from its classical role as a static regularization penalty. We investigate a fundamental question: *does weight decay stabilize training dynamics, and if so, through which mechanism?* Indeed, training stability is understood through different but related notions in the literature. We consider how weight decay affects the parameter-space dynamics and loss sharpness by analyzing its effects at the Edge of Stability (EoS). We show that weight decay robustly slows *progressive sharpening. Furthermore, we uncover a striking architecture-dependent phase transition. In CNNs, weight decay dampens the oscillations at the EoS, while in MLPs, increasing weight decay causes a phase transition in which the sharpness stabilizes at a threshold significantly below the theoretical frac2η boundary. We develop a mathematical framework that accurately models these phenomena and identify the global alignment of the parameter vector and the sharpness gradient as the mechanistic driver of the phase transition. Importantly, we show that these phenomena translate into stability in terms of search in function-space (NTK). Last, this shows that curvature thresholds obtained from convex/quadratic heuristics may not be reliable stability diagnostics under regularization.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 29 of its 40 references that are in the map. First posted on arXiv 15 May 2026.
Builds on
29 of 40 references.
References · 29
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2017Decoupled Weight Decay Regularizationcited by 1,722
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
- 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
- 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
- 2017L2 Regularization versus Batch and Weight Normalizationcited by 121
- 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
- 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
- 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 85
- 2018A Walk with SGDcited by 74
- 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
17 more
- 2024Understanding Optimization in Deep Learning with Central Flowscited by 57
- 2019Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergencecited by 54
- 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 50
- 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
- 2020On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspectivecited by 25
- 2024Edge of Stochastic Stability: Revisiting the Edge of Stability for SGDcited by 24
- 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
- 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
- 2022SGD and Weight Decay Secretly Minimize the Rank of Your Neural Networkcited by 17
- 2020Understanding Decoupled and Early Weight Decaycited by 12
- 2024Provable Emergence of Deep Neural Collapse and Low-Rank Bias in L2-Regularized Nonlinear Networkscited by 10
- 2025Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?cited by 10
- 2024How Neural Networks Learn the Support is an Implicit Regularization Effect of SGDcited by 8
- 2026Momentum Further Constrains Sharpness at the Edge of Stochastic Stabilitycited by 8
- 2026Non-Euclidean Gradient Descent Operates at the Edge of Stabilitycited by 7
- 2025Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stabilitycited by 6
- 2024Towards Better Generalization: Weight Decay Induces Low-rank Bias for Neural Networkscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.