mlsanityData to 14 Oct 2026

Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate

arXiv 2010.02916 ↗First posted 6 Oct 2020cs.LGcited by 59

Not on the map. Why?

Abstract (arXiv)

Recent works (e.g., (Li and Arora, 2020)) suggest that the use of popular normalization schemes (including Batch Normalization) in today's deep learning can move it far from a traditional optimization viewpoint, e.g., use of exponentially increasing learning rates. The current paper highlights other ways in which behavior of normalized nets departs from traditional viewpoints, and then initiates a formal framework for studying their mathematics via suitable adaptation of the conventional framework namely, modeling SGD-induced training trajectory via a suitable stochastic differential equation (SDE) with a noise term that captures gradient noise. This yields: (a) A new ' intrinsic learning rate' parameter that is the product of the normal learning rate and weight decay factor. Analysis of the SDE shows how the effective speed of learning varies and equilibrates over time under the control of intrinsic LR. (b) A challenge -- via theory and experiments -- to popular belief that good generalization requires large learning rates at the start of training. (c) New experiments, backed by mathematical intuition, suggesting the number of steps to equilibrium (in function space) scales as the inverse of the intrinsic learning rate, as opposed to the exponential time convergence bound implied by SDE analysis. We name it the Fast Equilibrium Conjecture and suggest it holds the key to why Batch Normalization is effective.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 8; 2023: 13; 2024: 9; 2025: 6; 2026: 12.

59 papers.

Sort:

By area:

  1. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  2. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 402
  3. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  4. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
  5. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
  6. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  7. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  8. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  9. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 72
  10. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  11. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  12. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
38 more
  1. 2024Normalization and effective learning rates in reinforcement learningcited by 58
  2. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 53
  3. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 50
  4. 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 43
  5. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 35
  6. 2022Robust Training of Neural Networks Using Scale Invariant Architecturescited by 29
  7. 2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
  8. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  9. 2024Analyzing & Reducing the Need for Learning Rate Warmup in GPT Trainingcited by 22
  10. 2021On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decaycited by 20
  11. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  12. 2021The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusioncited by 19
  13. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
  14. 2024Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descentcited by 19
  15. 2023TWINS: A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalizationcited by 18
  16. 2024Directional Smoothness and Gradient Methods: Convergence and Adaptivitycited by 17
  17. 2026Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimizationcited by 16
  18. 2021Neural Network Weights Do Not Converge to Stationary Points: An Invariant Measure Perspectivecited by 15
  19. 2023Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregressioncited by 14
  20. 2025Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetriescited by 14
  21. 2021Acceleration via Fractal Learning Rate Schedulescited by 13
  22. 2022Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysiscited by 13
  23. 2025Parameter Symmetry Potentially Unifies Deep Learning Theorycited by 12
  24. 2021How to decay your learning ratecited by 11
  25. 2022Maximal Initial Learning Rates in Deep ReLU Networkscited by 11
  26. 2023Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decaycited by 9
  27. 2024Understanding the Generalization Benefits of Late Learning Rate Decaycited by 6
  28. 2024Where Do Large Learning Rates Lead Us?cited by 6
  29. 2025On Learning Verifiers and Implications to Chain-of-Thought Reasoningcited by 6
  30. 2023Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networkscited by 5
  31. 2023Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescalingcited by 4
  32. 2024The Fair Language Model Paradoxcited by 4
  33. 2026Weight Decay Improves Language Model Plasticitycited by 4
  34. 2026Hyperball May Not Be a Free Lunchcited by 4
  35. 2023On the Training Instability of Shuffling SGD with Batch Normalizationcited by 3
  36. 2026Effective Learning Rate Governs Loss Dynamics in Language Model Pretrainingcited by 3
  37. 2026On the Nonlinearity of Learning Rate Scaling for LLM Trainingcited by 2
  38. 2023No Data Augmentation? Alternative Regularizations for Effective Training on Small Datasetscited by 1

50 most cited of 59 All 59 (JSON)