mlsanityData to 14 Oct 2026

Adaptive Gradient Methods at the Edge of Stability

arXiv 2207.14484 ↗First posted 29 Jul 2022cs.LGcited by 101

Not on the map. Why?

Abstract (arXiv)

Very little is known about the training dynamics of adaptive gradient methods like Adam in deep learning. In this paper, we shed light on the behavior of these algorithms in the full-batch and sufficiently large batch settings. Specifically, we empirically demonstrate that during full-batch training, the maximum eigenvalue of the preconditioned Hessian typically equilibrates at a certain numerical value -- the stability threshold of a gradient descent algorithm. For Adam with step size η and β1 = 0.9, this stability threshold is 38/η. Similar effects occur during minibatch training, especially as the batch size grows. Yet, even though adaptive methods train at the ``Adaptive Edge of Stability'' (AEoS), their behavior in this regime differs in a significant way from that of non-adaptive methods at the EoS. Whereas non-adaptive algorithms at the EoS are blocked from entering high-curvature regions of the loss landscape, adaptive gradient methods at the AEoS can keep advancing into high-curvature regions, while adapting the preconditioner to compensate. Our findings can serve as a foundation for the community's future understanding of adaptive gradient methods in deep learning.

Key facts. Cited by 101 papers in the map (data through 14 Oct 2026). Builds on 37 of its 53 references that are in the map. First posted on arXiv 29 Jul 2022.

Builds on

37 of 53 references.

References · 37

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  3. 2016Wide Residual Networkscited by 3,070
  4. 2020On Layer Normalization in the Transformer Architecturecited by 922
  5. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  6. 2019On the Convergence of Adam and Beyondcited by 841
  7. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  8. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  9. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  10. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  11. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
  12. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
25 more
  1. 2020Understanding the Difficulty of Training Transformerscited by 210
  2. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  3. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
  4. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  5. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 156
  6. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  7. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 139
  8. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 126
  9. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 125
  10. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
  11. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 90
  12. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 88
  13. 2022Understanding the unstable convergence of gradient descentcited by 86
  14. 2018Training Tips for the Transformer Modelcited by 85
  15. 2020Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalizationcited by 62
  16. 2018Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimizationcited by 49
  17. 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 46
  18. 2020Disentangling Adaptive Gradient Methods from Learning Ratescited by 42
  19. 2021A Loss Curvature Perspective on Training Instability in Deep Learningcited by 40
  20. 2020A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithmscited by 33
  21. 2019Linear Convergence of Adaptive Stochastic Gradient Descentcited by 29
  22. 2018A general system of differential equations to model first order adaptive algorithmscited by 28
  23. 2018Step Size Matters in Deep Learningcited by 19
  24. 2020Flatness is a False Friendcited by 19
  25. 2019At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?cited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 24; 2024: 18; 2025: 16; 2026: 40.

101 papers.

Sort:

By area:

  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  2. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  3. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
  4. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  5. 2023Stable and low-precision training for large-scale vision-language modelscited by 75
  6. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 72
  7. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  8. 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 53
  9. 2025Optimizing the Optimizer for Physics-Informed Neural Networks and Kolmogorov-Arnold Networkscited by 42
  10. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  11. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  12. 2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Trainingcited by 30
38 more
  1. 2023On the Implicit Bias of Adamcited by 27
  2. 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenoncited by 24
  3. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  4. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 23
  5. 2024Surge Phenomenon in Optimal Learning Rate and Batch Size Scalingcited by 22
  6. 2025Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networkscited by 22
  7. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 21
  8. 2023When and Why Momentum Accelerates SGD:An Empirical Studycited by 19
  9. 2024High dimensional analysis reveals conservative sharpening and a stochastic edge of stabilitycited by 18
  10. 2024FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher informationcited by 18
  11. 2023Sharpness-Aware Minimization and the Edge of Stabilitycited by 17
  12. 2024Directional Smoothness and Gradient Methods: Convergence and Adaptivitycited by 17
  13. 2023The Geometry of Neural Nets' Parameter Spaces Under Reparametrizationcited by 16
  14. 2025What Really Matters in Matrix-Whitening Optimizers?cited by 16
  15. 2026A Defense of the Quadratic Modelcited by 14
  16. 2023Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimizationcited by 13
  17. 2024Learning in PINNs: Phase transition, diffusion equilibrium, and generalizationcited by 13
  18. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  19. 2025Adaptive Preconditioners Trigger Loss Spikes in Adamcited by 13
  20. 2022Maximal Initial Learning Rates in Deep ReLU Networkscited by 11
  21. 2023mSAM: Micro-Batch-Averaged Sharpness-Aware Minimizationcited by 10
  22. 2025Representation Meets Optimization: Training PINNs and PIKANs for Gray-Box Discovery in Systems Pharmacologycited by 10
  23. 2026Momentum Further Constrains Sharpness at the Edge of Stochastic Stabilitycited by 10
  24. 2025Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Trainingcited by 9
  25. 2026Non-Euclidean Gradient Descent Operates at the Edge of Stabilitycited by 9
  26. 2023Benign Oscillation of Stochastic Gradient Descent with Large Learning Ratescited by 8
  27. 2024Stepping on the Edge: Curvature Aware Learning Rate Tunerscited by 8
  28. 2024No More Adam: Learning Rate Scaling at Initialization is All You Needcited by 8
  29. 2025GradPower: Powering Gradients for Faster Language Model Pre-Trainingcited by 8
  30. 2026A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMscited by 8
  31. 2024Exact Risk Curves of signSGD in High-Dimensions: Quantifying Preconditioning and Noise-Compression Effectscited by 7
  32. 2023The Inductive Bias of Flatness Regularization for Deep Matrix Factorizationcited by 6
  33. 2024Stable Language Model Pre-training by Reducing Embedding Variabilitycited by 6
  34. 2024Fast Deep Hedging with Second-Order Optimizationcited by 6
  35. 2025How Memory in Optimization Algorithms Implicitly Modifies the Losscited by 6
  36. 2026(How) Learning Rates Regulate Catastrophic Overtrainingcited by 6
  37. 2026Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfercited by 6
  38. 2023Combining Explicit and Implicit Regularization for Efficient Learning in Deep Networkscited by 4

50 most cited of 101 All 101 (JSON)