mlsanityData to 14 Oct 2026

SGD with Adaptive Preconditioning: Unified Analysis and Momentum Acceleration

arXiv 2506.23803 ↗First posted 30 Jun 2025cs.LG math.OCcited by 12

Not on the map. Why?

Abstract (arXiv)

In this paper, we revisit stochastic gradient descent (SGD) with AdaGrad-type preconditioning. Our contributions are twofold. First, we develop a unified convergence analysis of SGD with adaptive preconditioning under anisotropic or matrix smoothness and noise assumptions. This allows us to recover state-of-the-art convergence results for several popular adaptive gradient methods, including AdaGrad-Norm, AdaGrad, and ASGO/One-sided Shampoo. In addition, we establish the fundamental connection between two recently proposed algorithms, Scion and DASGO, and provide the first theoretical guarantees for the latter. Second, we show that the convergence of methods like AdaGrad and DASGO can be provably accelerated beyond the best-known rates using Nesterov momentum. Consequently, we obtain the first theoretical justification that AdaGrad-type algorithms can simultaneously benefit from both diagonal preconditioning and momentum, which may provide an ultimate explanation for the practical efficiency of Adam.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 30 Jun 2025.

Builds on

35 of 51 references.

References · 35

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019On the Convergence of Adam and Beyondcited by 841
  5. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  6. 2024Old Optimizer, New Norm: An Anthologycited by 193
  7. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
  8. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  9. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  10. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 156
  11. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
  12. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 133
23 more
  1. 2017Lower Bounds for Finding Stationary Points Icited by 126
  2. 2018Black-Box Reductions for Parameter-free Online Learning in Banach Spacescited by 125
  3. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  4. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  5. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 92
  6. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  7. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 72
  8. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  9. 2019UniXGrad: A Universal, Adaptive Algorithm with Optimal Guarantees for Constrained Optimizationcited by 56
  10. 2018Online Adaptive Methods, Universality and Accelerationcited by 51
  11. 2019SGD Converges to Global Minimum in Deep Learning via Star-convex Pathcited by 39
  12. 2019Near-Optimal Methods for Minimizing Star-Convex Functions and Beyondcited by 36
  13. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  14. 2025Structured Preconditioners in Adaptive Optimization: A Unified Analysiscited by 33
  15. 2021Parameter-free Stochastic Optimization of Variationally Coherent Functionscited by 27
  16. 2024AdaGrad under Anisotropic Smoothnesscited by 22
  17. 2022Sharp Analysis of Stochastic Optimization under Global Kurdyka-Lojasiewicz Inequalitycited by 20
  18. 2023Normalized Gradients for Allcited by 20
  19. 2017A Unified Approach to Adaptive Regularization in Online and Stochastic Optimizationcited by 19
  20. 2024Accelerated Parameter-Free Stochastic Optimizationcited by 15
  21. 2024Universality of AdaGrad Stepsizes for Stochastic Optimization: Inexact Oracle, Acceleration and Variance Reductioncited by 9
  22. 2024On Linear Convergence in Smooth Convex-Concave Bilinearly-Coupled Saddle-Point Optimization: Lower Bounds and Optimal Algorithmscited by 6
  23. 2025Incorporating Preconditioning into Accelerated Approaches: Theoretical Guarantees and Practical Improvementcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 7.

12 papers.

  1. 2025AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updatescited by 23
  2. 2025Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Ratescited by 8
  3. 2025Nesterov Finds GRAAL: Optimal and Adaptive Gradient Method for Convex Optimizationcited by 5
  4. 2025A Tale of Two Geometries: Adaptive Optimizers and Non-Euclidean Descentcited by 5
  5. 2025Provable Benefit of SignGD: A Minimal Model Under Heavy-Tailed Class Imbalancecited by 5
  6. 2026A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad and Muoncited by 2
  7. 2026A Polyak-Ruppert Central Limit Theorem for SA-Adam with Momentum and Non-Convergent Adaptive Preconditioningcited by 2
  8. 2026Optimal Projection-Free Adaptive SGD for Matrix Optimizationcited by 1
  9. 2026Deterministic Adam-Inspired Methods with Accelerated Convergence Ratecited by 1
  10. 2026Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimizationcited by 1
  11. 2026Stochastic Non-Smooth Convex Optimization with Unbounded Gradientscited by 0
  12. 2026Second-Moment Stochastic Approximation Methodscited by 0