mlsanityData to 14 Oct 2026

Continuous-Time Analysis of Adaptive Optimization and Normalization

arXiv 2411.05746 ↗First posted 8 Nov 2024cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Adaptive optimization algorithms, particularly Adam and its variant AdamW, are fundamental components of modern deep learning. However, their training dynamics lack comprehensive theoretical understanding, with limited insight into why common practices -- such as specific hyperparameter choices and normalization layers -- contribute to successful generalization. This work presents a continuous-time formulation of Adam and AdamW, facilitating a tractable analysis of training dynamics that can shed light on such practical questions. We theoretically derive a stable region for Adam's hyperparameters (β, γ) that ensures bounded updates, empirically verifying these predictions by observing unstable exponential parameter growth outside of this stable region. Furthermore, we theoretically justify the success of normalization layers by uncovering an implicit meta-adaptive effect of scale-invariant architectural components. This insight leads to an explicit optimizer, 2-Adam, which we generalize to k-Adam -- an optimizer that applies an adaptive normalization procedure k times, encompassing Adam (corresponding to k=1) and Adam with a normalization layer (corresponding to k=2). Overall, our continuous-time formulation of Adam facilitates a principled analysis, offering deeper understanding of optimal hyperparameter choices and architectural decisions in modern deep learning.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 29 of its 36 references that are in the map. First posted on arXiv 8 Nov 2024.

Builds on

29 of 36 references.

References · 29

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2016Layer Normalizationcited by 5,669
  4. 2019On the Convergence of Adam and Beyondcited by 841
  5. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  6. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  7. 2018How Does Batch Normalization Help Optimization?cited by 357
  8. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
  9. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 156
  10. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  11. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
  12. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 96
17 more
  1. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
  2. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  3. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
  4. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 51
  5. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 50
  6. 2018Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimizationcited by 49
  7. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 49
  8. 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 46
  9. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 39
  10. 2024The Implicit Bias of Adam on Separable Datacited by 36
  11. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  12. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  13. 2018A general system of differential equations to model first order adaptive algorithmscited by 28
  14. 2018Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimizationcited by 27
  15. 2021Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learningcited by 25
  16. 2024On Convergence of Adam for Stochastic Optimization under Relaxed Assumptionscited by 21
  17. 2019The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networkscited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026A Rod Flow Model for Adam at the Edge of Stabilitycited by 3
  2. 2026From Adam to Adam-Like Lagrangians: Second-Order Nonlocal Dynamicscited by 1
  3. 2026Deterministic Adam-Inspired Methods with Accelerated Convergence Ratecited by 1
  4. 2026Optimizer Geometry Sets the Pace: Spectral Learning Dynamics in Matrix Factorizationcited by 0