mlsanityData to 7 Oct 2026

A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad and Muon

arXiv 2604.17423 ↗First posted 19 Apr 2026cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

A unified framework for first-order optimization algorithms fornonconvex unconstrained optimization is proposed that uses adaptivelypreconditioned gradients and includes popular methods such as full anddiagonal AdaGrad, AdaNorm, as well as an adpative variant of Muon. This framework also allows combining heterogeneous geometries across different groups of variables while preserving a unified convergence analysis. A fully stochastic global rate-of-convergence analysis is conducted for all methods in the framework, with andwithout two types of momentum, using reasonable assumptions on the variance of the gradient oracle and without assuming bounded stochastic gradients or small enough stepsize.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 43 of its 57 references that are in the map. First posted on arXiv 19 Apr 2026.

Builds on

43 of 57 references.

References · 43

  1. 2021Convex optimizationcited by 2,479
  2. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  3. 2019On the Convergence of Adam and Beyondcited by 834
  4. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  5. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  6. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  7. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 179
  8. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  9. 2024Old Optimizer, New Norm: An Anthologycited by 173
  10. 2020A Simple Convergence Proof of Adam and Adagradcited by 169
  11. 2018Stochastic model-based minimization of weakly convex functionscited by 157
  12. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 155
31 more
  1. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 124
  2. 2022Adam Can Converge Without Any Modification On Update Rulescited by 112
  3. 2025A Note on the Convergence of Muoncited by 78
  4. 2022The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variancecited by 60
  5. 2025AdaMuon: Adaptive Muon Optimizercited by 59
  6. 2023Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptionscited by 55
  7. 2017Variants of RMSProp and Adagrad with Logarithmic Regret Boundscited by 50
  8. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
  9. 2023Beyond Uniform Smoothness: A Stopped Analysis of Adaptive SGDcited by 41
  10. 2022High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsizecited by 40
  11. 2020The Geometry of Sign Gradient Descentcited by 39
  12. 2023SGD with AdaGrad Stepsizes: Full Adaptivity with High Probability to Unknown Parameters, Unbounded Gradients and Affine Variancecited by 30
  13. 2025Structured Preconditioners in Adaptive Optimization: A Unified Analysiscited by 30
  14. 2025COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMscited by 28
  15. 2018WNGrad: Learn the Learning Rate in Gradient Descentcited by 24
  16. 2023Adam-family Methods for Nonsmooth Optimization with Convergence Guaranteescited by 22
  17. 2025AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updatescited by 22
  18. 2022Fully Stochastic Trust-Region Sequential Quadratic Programming for Equality-Constrained Optimization Problemscited by 21
  19. 2017A Unified Approach to Adaptive Regularization in Online and Stochastic Optimizationcited by 19
  20. 2024Revisiting Convergence of AdaGrad with Relaxed Assumptionscited by 14
  21. 2024Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variancecited by 14
  22. 2022On the Convergence of AdaGrad(Norm) on Rd: Beyond Convexity, Non-Asymptotic Rate and Accelerationcited by 13
  23. 2025SGD with Adaptive Preconditioning: Unified Analysis and Momentum Accelerationcited by 12
  24. 2017The duality structure gradient descent algorithm: analysis and applications to neural networkscited by 8
  25. 2019Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networkscited by 7
  26. 2022Convergence of First-Order Methods for Constrained Nonconvex Optimization with Dependent Datacited by 7
  27. 2022Complexity and performance for two classes of noise-tolerant first-order algorithmscited by 3
  28. 2023Divergence of the ADAM algorithm with fixed-stepsize: a (very) simple examplecited by 2
  29. 2026Projected Stochastic Momentum Methods for Nonlinear Equality-Constrained Optimization for Machine Learningcited by 1
  30. 2026An objective-function-free algorithm for general smooth constrained optimizationcited by 1
  31. 2026An objective-function-free algorithm for nonconvex stochastic optimization with deterministic equality and inequality constraintscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Stochastic convergence of parallel asynchronous adaptive first-order methodscited by 0
  2. 2026Two-level domain-decomposition AdaGrad method for scalable training of graph neural networkscited by 0