mlsanityData to 7 Oct 2026

Stochastic convergence of parallel asynchronous adaptive first-order methods

arXiv 2606.01787 ↗First posted 1 Jun 2026cs.AI math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

A new class of asynchronous adaptive first-order optimization methods is introduced, comprising asynchronous variants of several popular algorithms. Versions of these methods using momentum and/or inexact normalization are also considered. The convergence of methods in the class on non-convex functions is analyzed in a fully stochastic setting, and is shown to be (up to logarithmic factors) of order O(1/sqrt{t}) under reasonable assumptions. Numerical experiments suggest that such asynchronous adaptive algorithms are very relevant in heterogeneous large-scale machine learning systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 66 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

27 of 66 references.

References · 27

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  2. 2021Convex optimizationcited by 2,479
  3. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  4. 2025Muon is Scalable for LLM Trainingcited by 381
  5. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  6. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 179
  7. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  8. 2024Old Optimizer, New Norm: An Anthologycited by 173
  9. 2015Asynchronous Parallel Stochastic Gradient for Nonconvex Optimizationcited by 115
  10. 2018SGD and Hogwild! Convergence Without the Bounded Gradients Assumptioncited by 94
  11. 2015Perturbed Iterate Analysis for Asynchronous Stochastic Optimizationcited by 67
  12. 2022The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variancecited by 60
15 more
  1. 2025AdaMuon: Adaptive Muon Optimizercited by 59
  2. 2023Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptionscited by 55
  3. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
  4. 2018Improved asynchronous parallel optimization analysis for stochastic incremental methodscited by 30
  5. 2018New Convergence Aspects of Stochastic Gradient Algorithmscited by 30
  6. 2023SGD with AdaGrad Stepsizes: Full Adaptivity with High Probability to Unknown Parameters, Unbounded Gradients and Affine Variancecited by 30
  7. 2023Adam-family Methods for Nonsmooth Optimization with Convergence Guaranteescited by 22
  8. 2025AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updatescited by 22
  9. 2015ARock: an Algorithmic Framework for Asynchronous Parallel Coordinate Updatescited by 21
  10. 2021Asynchronous Iterations in Optimization: New Sequence Results and Sharper Algorithmic Guaranteescited by 17
  11. 2024Revisiting Convergence of AdaGrad with Relaxed Assumptionscited by 14
  12. 2022DADAO: Decoupled Accelerated Decentralized Asynchronous Optimizationcited by 8
  13. 2019Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networkscited by 7
  14. 2023A Block-Coordinate Approach of Multi-level Optimization with an Application to Physics-Informed Neural Networkscited by 7
  15. 2026A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad and Muoncited by 2

Built on by

Not cited yet.