mlsanityData to 7 Oct 2026

When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay

arXiv 2609.09116 ↗First posted 8 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Normalization renders large parts of neural networks effectively scale invariant, inducing a hidden feedback loop in which learning-rate schedules and weight decay interact through the parameter norm to control the effective step taken by the optimizer. We show that this interaction is governed by an exact discrete-time law: a single scalar quantity captures all schedule and decay forcing, while norm growth induces an opposing geometric self-quenching effect. This yields a sharp boundary that cleanly separates contraction- and expansion-dominated effective learning rate regimes. To understand the underlying mechanism, we provide exact analysis of a fully solved normalized regression model where the dynamics reduce to two dimensions and show that the balance point is intrinsically unstable, implying that constant learning rate with weight decay cannot stably maintain an interior equilibrium and instead produces recurrent behavior driven by discrete-time Jacobian structure. We further extend this perspective across optimizers through unified homogeneous-optimizer framework that reveals a structural dichotomy in self-quenching strength, providing a first-principles explanation for why adaptive methods exhibit systematically weaker stabilization under normalization. Across dynamical systems and neural networks (MLP, CNN, GPT2 / MNIST, CIFAR, wikiText, OpenWebText), the predicted law holds with high precision and enables direct control of training via the identified scalar, with performance peaking sharply at the predicted boundary. Together, these results isolate a single governing quantity for scale-invariant optimization, providing a precise and actionable lens on training dynamics, optimizer behavior, and schedule design in modern deep learning. Code is available in https://github.com/shasanamin/normalized-optimization-dynamics.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 35 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

28 of 35 references.

References · 28

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2016Pointer Sentinel Mixture Modelscited by 2,992
  3. 2018Group Normalizationcited by 1,514
  4. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  5. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  6. 2018Understanding Batch Normalizationcited by 157
  7. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  8. 2017L2 Regularization versus Batch and Weight Normalizationcited by 121
  9. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  10. 2018Three Mechanisms of Weight Decay Regularizationcited by 112
  11. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 93
  12. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
16 more
  1. 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 75
  2. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
  3. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  4. 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
  5. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  6. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  7. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 50
  8. 2017Riemannian approach to batch normalizationcited by 35
  9. 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 34
  10. 2019Online Normalization for Training Neural Networkscited by 29
  11. 2018Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimizationcited by 27
  12. 2025Why Gradients Rapidly Increase Near the End of Trainingcited by 21
  13. 2021On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decaycited by 20
  14. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
  15. 2020Spherical Perspective on Learning with Normalization Layerscited by 9
  16. 2025Large Stepsizes Accelerate Gradient Descent for Regularized Logistic Regressioncited by 7

Built on by

Not cited yet.