mlsanityData to 7 Oct 2026

Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks

arXiv 2305.17212 ↗First posted 26 May 2023cs.LGcited by 50

Not on the map. Why?

Abstract (arXiv)

This study investigates how weight decay affects the update behavior of individual neurons in deep neural networks through a combination of applied analysis and experimentation. Weight decay can cause the expected magnitude and angular updates of a neuron's weight vector to converge to a steady state we call rotational equilibrium. These states can be highly homogeneous, effectively balancing the average rotation -- a proxy for the effective learning rate -- across different layers and neurons. Our work analyzes these dynamics across optimizers like Adam, Lion, and SGD with momentum, offering a new simple perspective on training that elucidates the efficacy of widely used but poorly understood methods in deep learning. We demonstrate how balanced rotation plays a key role in the effectiveness of normalization like Weight Standardization, as well as that of AdamW over Adam with L2-regularization. Finally, we show that explicitly controlling the rotation provides the benefits of weight decay while substantially reducing the need for learning rate warmup.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 39 of its 66 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

39 of 66 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2016Layer Normalizationcited by 5,579
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  12. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
27 more
  1. 2018Group Normalizationcited by 1,514
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  3. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  4. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  5. 2017Large Batch Training of Convolutional Networkscited by 529
  6. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  7. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  8. 2018An Empirical Model of Large-Batch Trainingcited by 268
  9. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  10. 2017L2 Regularization versus Batch and Weight Normalizationcited by 121
  11. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  12. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 117
  13. 2018Three Mechanisms of Weight Decay Regularizationcited by 112
  14. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 100
  15. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 87
  16. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  17. 2018Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutescited by 63
  18. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  19. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  20. 2019Micro-Batch Training with Batch-Channel Normalization and Weight Standardizationcited by 45
  21. 2019Online Normalization for Training Neural Networkscited by 29
  22. 2022Robust Training of Neural Networks Using Scale Invariant Architecturescited by 29
  23. 2020On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspectivecited by 25
  24. 2021Learning by Turning: Neural Architecture Aware Optimisationcited by 23
  25. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
  26. 2017Projection Based Weight Normalization for Deep Neural Networkscited by 8
  27. 2021FixNorm: Dissecting Weight Decay for Training Deep Neural Networkscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 16; 2026: 26.

50 papers.

Sort:

By area:

  1. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  2. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
  3. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 68
  4. 2024How Does Critical Batch Size Scale in Pre-training?cited by 68
  5. 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 42
  6. 2025Grokking at the Edge of Numerical Stabilitycited by 41
  7. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 41
  8. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 38
  9. 2024Analyzing & Reducing the Need for Learning Rate Warmup in GPT Trainingcited by 22
  10. 2025Why Gradients Rapidly Increase Near the End of Trainingcited by 21
  11. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  12. 2024Learning in PINNs: Phase transition, diffusion equilibrium, and generalizationcited by 13
38 more
  1. 2025AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMscited by 12
  2. 2025Cautious Weight Decaycited by 12
  3. 2026Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimizationcited by 12
  4. 2025Towards Fully FP8 GEMM LLM Training at Scalecited by 11
  5. 2025Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Schedulercited by 9
  6. 2025Taming Transformer Without Using Learning Rate Warmupcited by 8
  7. 2025Robust Layerwise Scaling Rules by Proper Weight Decay Tuningcited by 8
  8. 2025Better Convergence Guarantees for Sign-Based Momentum Methodscited by 6
  9. 2026ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Modelscited by 6
  10. 2026How to Set the Learning Rate for Large-Scale Pre-training?cited by 5
  11. 2024Pyramid Vector Quantization for LLMscited by 4
  12. 2025Efficient Hyperparameter Tuning via Trajectory Invariance Principlecited by 4
  13. 2026Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizerscited by 4
  14. 2026Weight Decay Improves Language Model Plasticitycited by 4
  15. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 4
  16. 2026Muown Implicitly Performs Angular Step-size Decaycited by 3
  17. 2026Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectorscited by 3
  18. 2026Learnable Multipliers: Freeing the Scale of Language Model Matrix Layerscited by 2
  19. 2026Disentangling Geometry, Performance, and Training in Language Modelscited by 2
  20. 2026Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamicscited by 2
  21. 2026On the Nonlinearity of Learning Rate Scaling for LLM Trainingcited by 2
  22. 2026Scale Weight Decay and Train Bettercited by 2
  23. 2026Effective Learning Rate Governs Loss Dynamics in Language Model Pretrainingcited by 2
  24. 2025Energy Considerations for Large Pretrained Neural Networkscited by 1
  25. 2025Correction of Decoupled Weight Decaycited by 1
  26. 2026Demystifying Manifold Constraints in LLM Pre-trainingcited by 1
  27. 2026Does Weight Decay Enhance Training Stability?cited by 1
  28. 2026SOAP, Muon, and Beyond: Pushing LLM Pretraining Scalescited by 1
  29. 2026Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Trainingcited by 1
  30. 2025Can Stationary Distributions of Scale-Invariant Neural Networks Be Described by the Thermodynamics of an Ideal Gas?cited by 0
  31. 2026Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuningcited by 0
  32. 2026Training nGPTcited by 0
  33. 2026RODE: A Radial-Orthogonal Decoupled Engine for Optimizationcited by 0
  34. 2026Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretrainingcited by 0
  35. 2026HyperTransfer: Understanding the Equivalence between Base Optimizer and Hyperballcited by 0
  36. 2026When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decaycited by 0
  37. 2026A Mesoscopic View of Transformer Weights Through Row and Column Scale Fieldscited by 0
  38. 2026The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidationcited by 0