mlsanityData to 14 Oct 2026

Stochastic Approximation with Block Coordinate Optimal Stepsizes

arXiv 2507.08963 ↗First posted 11 Jul 2025math.OC cs.LG stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

We consider stochastic approximation with block-coordinate stepsizes and propose adaptive stepsize rules that aim to minimize the expected distance from the next iterate to an (unknown) target point. These stepsize rules employ online estimates of the second moment of the search direction along each block coordinate. The popular Adam algorithm can be interpreted as a variant with a specific estimator. By leveraging a simple conditional estimator, we derive a new method that obtains competitive performance against Adam but requires less memory and fewer hyper-parameters. We prove that this family of methods converges almost surely to a small neighborhood of the target point, and the radius of the neighborhood depends on the bias and variance of the second-moment estimator. Our analysis relies on a simple aiming condition that assumes neither convexity nor smoothness, thus has broad applicability.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 24 of its 55 references that are in the map. First posted on arXiv 11 Jul 2025.

Builds on

24 of 55 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2019On the Convergence of Adam and Beyondcited by 841
  6. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  7. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  8. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 202
  9. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  10. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  11. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 139
  12. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
12 more
  1. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 126
  2. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  3. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 80
  4. 2019Stochastic Sign Descent Methods: New Algorithms and Better Theorycited by 49
  5. 2023Optimal Linear Decay Learning Rate Schedules and Further Refinementscited by 43
  6. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  7. 2023Effective Theory of Transformers at Initializationcited by 21
  8. 2018An improvement of the convergence proof of the ADAM-Optimizercited by 19
  9. 2024FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher informationcited by 18
  10. 2024AdaFisher: Adaptive Second Order Optimization via Fisher Informationcited by 14
  11. 2024Efficient Sign-Based Optimization: Accelerating Convergence via Variance Reductioncited by 13
  12. 2025AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-trainingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectorycited by 8
  2. 2026Schattor: Schatten-family methods for deep learning optimizationcited by 1
  3. 2026Second-Moment Stochastic Approximation Methodscited by 0