# On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective

arXiv: https://arxiv.org/abs/2011.11152 · HTML: https://mlsanity.com/paper/2011.11152/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Nov 2020.

## Built on by (most cited first, 25 of 25)

- [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://mlsanity.com/paper/2407.19669/) (2024)
- [Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks](https://mlsanity.com/paper/2305.17212/) (2023)
- [Weight decay induces low-rank attention layers](https://mlsanity.com/paper/2410.23819/) (2024)
- [QVGen: Pushing the Limit of Quantized Video Generative Models](https://mlsanity.com/paper/2505.11497/) (2025)
- [Why Gradients Rapidly Increase Near the End of Training](https://mlsanity.com/paper/2506.02285/) (2025)
- [Generalized Gradient Norm Clipping & Non-Euclidean (L0,L1)-Smoothness](https://mlsanity.com/paper/2506.01913/) (2025)
- [On the Overlooked Structure of Stochastic Gradients](https://mlsanity.com/paper/2212.02083/) (2022)
- [AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs](https://mlsanity.com/paper/2506.14562/) (2025)
- [Cautious Weight Decay](https://mlsanity.com/paper/2510.12402/) (2025)
- [Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?](https://mlsanity.com/paper/2504.12883/) (2025)
- [Do we really have to filter out random noise in pre-training data for language models?](https://mlsanity.com/paper/2502.06604/) (2025)
- [DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models](https://mlsanity.com/paper/2410.09344/) (2024)
- [Robust Layerwise Scaling Rules by Proper Weight Decay Tuning](https://mlsanity.com/paper/2510.15262/) (2025)
- [Residual-Concatenate Neural Network with Deep Regularization Layers for Binary Classification](https://mlsanity.com/paper/2205.12775/) (2022)
- [Gradient Correction in Federated Learning with Adaptive Optimization](https://mlsanity.com/paper/2502.02727/) (2025)
- [NeuralGrok: Accelerate Grokking by Neural Gradient Transformation](https://mlsanity.com/paper/2504.17243/) (2025)
- [On effects of Knowledge Distillation on Transfer Learning](https://mlsanity.com/paper/2210.09668/) (2022)
- [Neural Networks with (Low-Precision) Polynomial Approximations: New Insights and Techniques for Accuracy Improvement](https://mlsanity.com/paper/2402.11224/) (2024)
- [Correction of Decoupled Weight Decay](https://mlsanity.com/paper/2512.08217/) (2025)
- [Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Better](https://mlsanity.com/paper/2602.05393/) (2026)
- [Learning to Forget: Continual Learning with Adaptive Weight Decay](https://mlsanity.com/paper/2604.27063/) (2026)
- [Demystifying Manifold Constraints in LLM Pre-training](https://mlsanity.com/paper/2605.04418/) (2026)
- [Does Weight Decay Enhance Training Stability?](https://mlsanity.com/paper/2605.16622/) (2026)
- [Neural Field Classifiers via Target Encoding and Classification Loss](https://mlsanity.com/paper/2403.01058/) (2024)
- [Self Identity Mapping](https://mlsanity.com/paper/2509.18165/) (2025)

## Abstract (arXiv)

Weight decay is a simple yet powerful regularization technique that has been very widely used in training of deep neural networks (DNNs). While weight decay has attracted much attention, previous studies fail to discover some overlooked pitfalls on large gradient norms resulted by weight decay. In this paper, we discover that, weight decay can unfortunately lead to large gradient norms at the final phase (or the terminated solution) of training, which often indicates bad convergence and poor generalization. To mitigate the gradient-norm-centered pitfalls, we present the first practical scheduler for weight decay, called the Scheduled Weight Decay (SWD) method that can dynamically adjust the weight decay strength according to the gradient norm and significantly penalize large gradient norms during training. Our experiments also support that SWD indeed mitigates large gradient norms and often significantly outperforms the conventional constant weight decay strategy for Adaptive Moment Estimation (Adam).