mlsanityData to 14 Oct 2026

Stochastic Adaptive Gradient Descent Without Descent

arXiv 2509.14969 ↗First posted 18 Sep 2025cs.LG math.OC stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce a new adaptive step-size strategy for convex optimization with stochastic gradient that exploits the local geometry of the objective function only by means of a first-order stochastic oracle and without any hyper-parameter tuning. The method comes from a theoretically-grounded adaptation of the Adaptive Gradient Descent Without Descent method to the stochastic setting. We prove the convergence of stochastic gradient descent with our step-size under various assumptions, and we show that it empirically competes against tuned baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 56 references that are in the map. First posted on arXiv 18 Sep 2025.

Builds on

33 of 56 references.

References · 33

  1. 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
  2. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 156
  3. 2020Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergencecited by 137
  4. 2016Coin Betting and Parameter-Free Online Learningcited by 133
  5. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 133
  6. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  7. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  8. 2019Adaptive Gradient Descent without Descentcited by 79
  9. 2018Stochastic (Approximate) Proximal Point Methods: Convergence, Optimality, and Adaptivitycited by 61
  10. 2022The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variancecited by 61
  11. 2019Revisiting the Polyak step sizecited by 55
  12. 2022Making SGD Parameter-Freecited by 50
21 more
  1. 2022A Neural-Network-Based Convex Regularizer for Inverse Problemscited by 41
  2. 2017Online to Offline Conversions, Universality and Adaptive Minibatch Sizescited by 39
  3. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  4. 2016Barzilai-Borwein Step Size for Stochastic Gradient Descentcited by 33
  5. 2014Simultaneous Model Selection and Optimization through Parameter-free Stochastic Learningcited by 27
  6. 2021Parameter-free Stochastic Optimization of Variationally Coherent Functionscited by 27
  7. 2023Adaptive Proximal Gradient Method for Convex Optimizationcited by 21
  8. 2021Convergence Rates of Gradient Methods for Convex Optimization in the Space of Measurescited by 20
  9. 2023Normalized Gradients for Allcited by 20
  10. 2023Adaptive proximal algorithms for convex optimization under local Lipschitz continuity of the gradientcited by 19
  11. 2023A simple uniformly optimal method without line search for convex optimizationcited by 18
  12. 2024Tuning-Free Stochastic Optimizationcited by 13
  13. 2022Grad-GradaGrad? A Non-Monotone Adaptive Stochastic Gradient Methodcited by 9
  14. 2022On Optimal Universal First-Order Methods for Minimizing Heterogeneous Sumscited by 9
  15. 2023Optimal and parameter-free gradient minimization methods for convex and nonconvex optimizationcited by 8
  16. 2020Complexity Guarantees for Polyak Steps with Momentumcited by 7
  17. 2023Statistical Complexity and Optimal Algorithms for Non-linear Ridge Banditscited by 7
  18. 2023Parameter-Free FISTA by Adaptive Restart and Backtrackingcited by 4
  19. 2021Second-order step-size tuning of SGD for non-convex optimizationcited by 3
  20. 2022Incorporating the Barzilai-Borwein Adaptive Step Size into Sugradient Methods for Deep Network Trainingcited by 1
  21. 2023Near-optimal Closed-loop Method via Lyapunov Damping for Convex Optimizationcited by 1

Built on by

Not cited yet.