mlsanityData to 14 Oct 2026

SANIA: Polyak-type Optimization Framework Leads to Scale Invariant Stochastic Algorithms

arXiv 2312.17369 ↗First posted 28 Dec 2023cs.LG math.OCcited by 5

Not on the map. Why?

Abstract (arXiv)

Adaptive optimization methods are widely recognized as among the most popular approaches for training Deep Neural Networks (DNNs). Techniques such as Adam, AdaGrad, and AdaHessian utilize a preconditioner that modifies the search direction by incorporating information about the curvature of the objective function. However, despite their adaptive characteristics, these methods still require manual fine-tuning of the step-size. This, in turn, impacts the time required to solve a particular problem. This paper presents an optimization framework named SANIA to tackle these challenges. Beyond eliminating the need for manual step-size hyperparameter settings, SANIA incorporates techniques to address poorly scaled or ill-conditioned problems. We also explore several preconditioning methods, including Hutchinson's method, which approximates the Hessian diagonal of the loss function. We conclude with an extensive empirical examination of the proposed techniques across classification tasks, covering both convex and non-convex contexts.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 37 of its 81 references that are in the map. First posted on arXiv 28 Dec 2023.

Builds on

37 of 81 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  6. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  7. 2019On the Convergence of Adam and Beyondcited by 841
  8. 2016Automatic chemical design using a data-driven continuous representation of moleculescited by 765
  9. 2016Neural Combinatorial Optimization with Reinforcement Learningcited by 608
  10. 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 417
  11. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 202
  12. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 156
25 more
  1. 2020Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergencecited by 137
  2. 2016Coin Betting and Parameter-Free Online Learningcited by 133
  3. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 133
  4. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  5. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  6. 2022Making SGD Parameter-Freecited by 50
  7. 2014Unconstrained Online Linear Learning in Hilbert Spaces: Minimax Algorithms and Normal Approximationscited by 46
  8. 2022Understanding AdamW through Proximal Methods and Scale-Freenesscited by 46
  9. 2019Training Neural Networks for and by Interpolationcited by 45
  10. 2022Dynamics of SGD with Stochastic Polyak Stepsizes: Truly Adaptive Variants and Convergence to Exact Solutioncited by 40
  11. 2023MoMo: Momentum Models for Adaptive Learning Ratescited by 24
  12. 2023Adaptive SGD with Polyak stepsize and Line-search: Robust Convergence and Variance Reductioncited by 23
  13. 2021Stochastic Mirror Descent: Convergence Analysis and Adaptive Variants via the Mirror Stochastic Polyak Stepsizecited by 22
  14. 2021Regularized Newton Method with Global O(1/k2) Convergencecited by 22
  15. 2022Super-Universal Regularized Newton Methodcited by 20
  16. 2023Mechanic: A Learning Rate Tunercited by 20
  17. 2019Adaptive scale-invariant online algorithms for learning linear modelscited by 19
  18. 2021Stochastic Polyak Stepsize with a Moving Targetcited by 19
  19. 2022Cutting Some Slack for SGD with Adaptive Polyak Stepsizescited by 18
  20. 2021Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order Informationcited by 16
  21. 2021Gradient Regularization of Newton Method with Bregman Distancescited by 16
  22. 2022SP2: A Second Order Stochastic Polyak Methodcited by 13
  23. 2019Stochastic Gradient Descent with Polyak's Learning Ratecited by 11
  24. 2023Function Value Learning: Adaptive Learning Rates Based on the Polyak Stepsize and Function Splitting in ERMcited by 8
  25. 2023Stochastic Gradient Descent with Preconditioned Polyak Step-sizecited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 1.

5 papers.

  1. 2024Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performancecited by 13
  2. 2025Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methodscited by 11
  3. 2024Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGradcited by 8
  4. 2024AdaBatchGrad: Combining Adaptive Batch Size and Adaptive Step Sizecited by 3
  5. 2026Byzantine-Robust Optimization under (L0, L1)-Smoothnesscited by 0