mlsanityData to 13 Oct 2026

Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad

arXiv 2403.02648 ↗First posted 5 Mar 2024cs.LG cs.AI math.OCcited by 8

Not on the map. Why?

Abstract (arXiv)

Adaptive methods are extremely popular in machine learning as they make learning rate tuning less expensive. This paper introduces a novel optimization algorithm named KATE, which presents a scale-invariant adaptation of the well-known AdaGrad algorithm. We prove the scale-invariance of KATE for the case of Generalized Linear Models. Moreover, for general smooth non-convex problems, we establish a convergence rate of O left(fraclog T√T right) for KATE, matching the best-known ones for AdaGrad and Adam. We also compare KATE to other state-of-the-art adaptive algorithms Adam and AdaGrad in numerical experiments with different problems, including complex machine learning tasks like image classification and text classification on real data. The results indicate that KATE consistently outperforms AdaGrad and matches/surpasses the performance of Adam in all considered scenarios.

Key facts. Cited by 8 papers in the map (data through 13 Oct 2026). Builds on 35 of its 56 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

35 of 56 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  4. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,627
  5. 2019On the Convergence of Adam and Beyondcited by 837
  6. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 335
  7. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 306
  8. 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 220
  9. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  10. 2020A Simple Convergence Proof of Adam and Adagradcited by 169
  11. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 155
  12. 2020Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergencecited by 137
23 more
  1. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 132
  2. 2017Lower Bounds for Finding Stationary Points Icited by 125
  3. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  4. 2018SGD and Hogwild! Convergence Without the Bounded Gradients Assumptioncited by 95
  5. 2016Scale-Free Online Learningcited by 84
  6. 2022The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variancecited by 61
  7. 2020A Better Alternative to Error Feedback for Communication-Efficient Distributed Learningcited by 42
  8. 2022Dynamics of SGD with Stochastic Polyak Stepsizes: Truly Adaptive Variants and Convergence to Exact Solutioncited by 40
  9. 2015Scale-Free Algorithms for Online Linear Optimizationcited by 30
  10. 2019Linear Convergence of Adaptive Stochastic Gradient Descentcited by 29
  11. 2021Stochastic Mirror Descent: Convergence Analysis and Adaptive Variants via the Mirror Stochastic Polyak Stepsizecited by 22
  12. 2017Stochastic Recursive Gradient Algorithm for Nonconvex Optimizationcited by 20
  13. 2021Stochastic Polyak Stepsize with a Moving Targetcited by 19
  14. 2013A Generalized Online Mirror Descent with Applications to Classification and Regressioncited by 18
  15. 2021AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methodscited by 15
  16. 2022SP2: A Second Order Stochastic Polyak Methodcited by 13
  17. 2013Mini-Batch Primal and Dual Methods for SVMscited by 12
  18. 2018Inexact SARAH Algorithm for Stochastic Optimizationcited by 12
  19. 2019Stochastic Gradient Descent with Polyak's Learning Ratecited by 11
  20. 2021Random-reshuffled SARAH does not need a full gradient computationscited by 7
  21. 2023Stochastic Gradient Descent with Preconditioned Polyak Step-sizecited by 7
  22. 2023SANIA: Polyak-type Optimization Framework Leads to Scale Invariant Stochastic Algorithmscited by 5
  23. 2018Dual Free Adaptive Minibatch SDCA for Empirical Risk Minimizationcited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 4.

8 papers.

  1. 2024Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performancecited by 13
  2. 2025Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methodscited by 10
  3. 2025Safeguarded Stochastic Polyak Step Sizes for Non-smooth Optimization: Robust Performance Without Small (Sub)Gradientscited by 2
  4. 2026Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameterscited by 2
  5. 2025Dynamic Low-rank Approximation of Full-Matrix Preconditioner for Training Generalized Linear Modelscited by 1
  6. 2026AdaGrad-Diff: A New Version of the Adaptive Gradient Algorithmcited by 1
  7. 2026Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Schedulercited by 1
  8. 2026HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalizationcited by 0