mlsanityData to 14 Oct 2026

On the SDEs and Scaling Rules for Adaptive Gradient Algorithms

arXiv 2205.10287 ↗First posted 20 May 2022cs.LGcited by 104

Not on the map. Why?

Abstract (arXiv)

Approximating Stochastic Gradient Descent (SGD) as a Stochastic Differential Equation (SDE) has allowed researchers to enjoy the benefits of studying a continuous optimization trajectory while carefully preserving the stochasticity of SGD. Analogous study of adaptive gradient methods, such as RMSprop and Adam, has been challenging because there were no rigorously proven SDE approximations for these methods. This paper derives the SDE approximations for RMSprop and Adam, giving theoretical guarantees of their correctness as well as experimental validation of their applicability to common large-scaling vision and language settings. A key practical result is the derivation of a textitsquare root scaling rule to adjust the optimization hyperparameters of RMSprop and Adam when changing batch size, and its empirical validation in deep learning settings.

Key facts. Cited by 104 papers in the map (data through 14 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 20 May 2022.

Builds on

32 of 52 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  8. 2016Pointer Sentinel Mixture Modelscited by 3,096
  9. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  10. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  11. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  12. 2019On the Convergence of Adam and Beyondcited by 841
20 more
  1. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  2. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  3. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  4. 2017Three Factors Influencing Minima in SGDcited by 273
  5. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  6. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
  7. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  8. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 126
  9. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  10. 2022Should You Mask 15% in Masked Language Modeling?cited by 118
  11. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
  12. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 96
  13. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 90
  14. 2021How to Train BERT with an Academic Budgetcited by 80
  15. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  16. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  17. 2020Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Trainingcited by 50
  18. 2018Quasi-hyperbolic momentum and Adam for deep learningcited by 48
  19. 2020Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentumcited by 40
  20. 2020A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithmscited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 11; 2024: 16; 2025: 35; 2026: 38.

104 papers.

Sort:

By area:

  1. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  2. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
  3. 2025REVE: A Foundation Model for EEG -- Adapting to Any Setup with Large-Scale Pretraining on 25,000 Subjectscited by 80
  4. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 72
  5. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
  6. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 53
  7. 2024On Scaling Up 3D Gaussian Splatting Trainingcited by 49
  8. 2024Falcon Mamba: The First Competitive Attention-free 7B Language Modelcited by 46
  9. 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 43
  10. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 43
  11. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 41
  12. 2025In Search of Adam's Secret Saucecited by 40
38 more
  1. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 37
  2. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 35
  3. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  4. 2023On the Implicit Bias of Adamcited by 27
  5. 2025Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Durationcited by 27
  6. 2023How to Scale Your EMAcited by 24
  7. 2025Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Lawcited by 24
  8. 2025Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Trainingcited by 23
  9. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  10. 2024Analyzing & Reducing the Need for Learning Rate Warmup in GPT Trainingcited by 22
  11. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  12. 2025Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networkscited by 22
  13. 2023The Marginal Value of Momentum for Small Learning Rate SGDcited by 18
  14. 2025Is your batch size the problem? Revisiting the Adam-SGD gap in language modelingcited by 15
  15. 2023Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregressioncited by 14
  16. 2024AdaFisher: Adaptive Second Order Optimization via Fisher Informationcited by 14
  17. 2026A Defense of the Quadratic Modelcited by 14
  18. 2024Falcon2-11B Technical Reportcited by 12
  19. 2025Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Schedulercited by 12
  20. 2022A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimizationcited by 11
  21. 2023Stochastic Modified Equations and Dynamics of Dropout Algorithmcited by 11
  22. 2026Representation Distribution Matching for One-Step Visual Generationcited by 11
  23. 2025ODE approximation for the Adam algorithm: General and overparametrized settingcited by 10
  24. 2026Deriving Hyperparameter Scaling Laws via Modern Optimization Theorycited by 10
  25. 2024Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limitcited by 9
  26. 2025Optimal Scaling Needs Optimal Normcited by 8
  27. 2024Exact Risk Curves of signSGD in High-Dimensions: Quantifying Preconditioning and Noise-Compression Effectscited by 7
  28. 2026Optimizer Memory Makes Shuffle Order a First-Order Source of Fine-Tuning Noisecited by 7
  29. 2024SDEs for Minimax Optimizationcited by 6
  30. 2025How Memory in Optimization Algorithms Implicitly Modifies the Losscited by 6
  31. 2025Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Schedulingcited by 6
  32. 2025Relative Scaling Laws for LLMscited by 6
  33. 2026The Design Space of Tri-Modal Masked Diffusion Modelscited by 6
  34. 2026SOAP, Muon, and Beyond: Pushing LLM Pretraining Scalescited by 6
  35. 2023A Quadratic Synchronization Rule for Distributed Deep Learningcited by 5
  36. 2024Batch size invariant Adamcited by 5
  37. 2025Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEscited by 5
  38. 2025Scalable GANs with Transformerscited by 5

50 most cited of 104 All 104 (JSON)