mlsanityData to 7 Oct 2026

Grad-GradaGrad? A Non-Monotone Adaptive Stochastic Gradient Method

arXiv 2206.06900 ↗First posted 14 Jun 2022cs.LG math.OC stat.MLcited by 9

Not on the map. Why?

Abstract (arXiv)

The classical AdaGrad method adapts the learning rate by dividing by the square root of a sum of squared gradients. Because this sum on the denominator is increasing, the method can only decrease step sizes over time, and requires a learning rate scaling hyper-parameter to be carefully tuned. To overcome this restriction, we introduce GradaGrad, a method in the same family that naturally grows or shrinks the learning rate based on a different accumulation in the denominator, one that can both increase and decrease. We show that it obeys a similar convergence rate as AdaGrad and demonstrate its non-monotone adaptation capability with experiments.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 14 of its 28 references that are in the map. First posted on arXiv 14 Jun 2022.

Builds on

14 of 28 references.

References · 14

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019On the Convergence of Adam and Beyondcited by 834
  3. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  4. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 155
  5. 2020Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergencecited by 137
  6. 2019Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Ratescited by 119
  7. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 117
  8. 2019Adaptive Gradient Descent without Descentcited by 79
  9. 2016MetaGrad: Multiple Learning Rates in Online Learningcited by 46
  10. 2021Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimizationcited by 27
  11. 2019A Universal Algorithm for Variational Inequalities Adaptive to Smoothness and Noisecited by 26
  12. 2021MetaGrad: Adaptation using Multiple Learning Rates in Online Learningcited by 26
2 more
  1. 2020Statistical Adaptive Stochastic Gradient Methodscited by 8
  2. 2021Adaptive first-order methods revisited: Convex optimization without Lipschitz requirementscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 1; 2025: 2; 2026: 2.

9 papers.

  1. 2023Adaptive proximal algorithms for convex optimization under local Lipschitz continuity of the gradientcited by 19
  2. 2023Searching for Optimal Per-Coordinate Step-sizes with Multidimensional Backtrackingcited by 9
  3. 2023Adaptive Federated Learning with Auto-Tuned Clientscited by 7
  4. 2025Nesterov Finds GRAAL: Optimal and Adaptive Gradient Method for Convex Optimizationcited by 4
  5. 2024Adaptive proximal gradient methods are universal without approximationcited by 3
  6. 2023On the convergence of adaptive first order methods: proximal gradient and alternating minimization algorithmscited by 2
  7. 2025Stochastic Adaptive Gradient Descent Without Descentcited by 0
  8. 2026Stochastic Non-Smooth Convex Optimization with Unbounded Gradientscited by 0
  9. 2026Low-Rank Dependence Decomposition via Accelerated Symmetric Non-negative Matrix Factorizationcited by 0