mlsanityData to 14 Oct 2026

On the Algorithmic Stability and Generalization of Adaptive Optimization Methods

arXiv 2211.03970 ↗First posted 8 Nov 2022cs.LG math.OCcited by 5

Not on the map. Why?

Abstract (arXiv)

Despite their popularity in deep learning and machine learning in general, the theoretical properties of adaptive optimizers such as Adagrad, RMSProp, Adam or AdamW are not yet fully understood. In this paper, we develop a novel framework to study the stability and generalization of these optimization methods. Based on this framework, we show provable guarantees about such properties that depend heavily on a single parameter β2. Our empirical experiments support our claims and provide practical insights into the stability and generalization properties of adaptive optimization methods.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 27 of its 51 references that are in the map. First posted on arXiv 8 Nov 2022.

Builds on

27 of 51 references.

References · 27

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2019On the Convergence of Adam and Beyondcited by 841
  4. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  5. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  6. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 303
  7. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
  8. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  9. 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 119
  10. 2020Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descentcited by 119
  11. 2018On the importance of single directions for generalizationcited by 99
  12. 2019High probability generalization bounds for uniformly stable algorithms with nearly optimal ratecited by 98
15 more
  1. 2019Sharper bounds for uniformly stable algorithmscited by 93
  2. 2017Data-Dependent Stability of Stochastic Gradient Descentcited by 92
  3. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
  4. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 90
  5. 2018Stability and Convergence Trade-off of Iterative Optimization Algorithmscited by 53
  6. 2017Algorithmic stability and hypothesis complexitycited by 39
  7. 2018Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov accelerationcited by 39
  8. 2020Information-Theoretic Generalization Bounds for Meta-Learning and Applicationscited by 32
  9. 2019Hypothesis Set Stability and Generalizationcited by 23
  10. 2018Generalization Error Bounds with Probabilistic Guarantee for SGD in Nonconvex Optimizationcited by 14
  11. 2020Stochastic optimization with momentum: convergence, fluctuations, and traps avoidancecited by 10
  12. 2017Fast Rates for Empirical Risk Minimization of Strict Saddle Problemscited by 7
  13. 2020Adaptive First-and Zeroth-order Methods for Weakly Convex Stochastic Optimization Problemscited by 7
  14. 2020Convergence of adaptive algorithms for weakly convex constrained optimizationcited by 6
  15. 2019Calibrating the Adaptive Learning Rate to Improve Convergence of ADAMcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 0; 2025: 1; 2026: 2.

5 papers.

  1. 2023Generalization error bounds for iterative learning algorithms with bounded updatescited by 1
  2. 2023Signal Processing Meets SGD: From Momentum to Filtercited by 1
  3. 2026Dynamic Momentum Recalibration in Online Gradient Learningcited by 1
  4. 2025Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapescited by 0
  5. 2026Uniform Stability and Generalization Error of GD and SGD on Fixed-Point Parameterscited by 0