mlsanityData to 7 Oct 2026

Revisiting the Polyak step size

arXiv 1905.00313 ↗First posted 1 May 2019math.OCcited by 55

Not on the map. Why?

Abstract (arXiv)

This paper revisits the Polyak step size schedule for convex optimization problems, proving that a simple variant of it simultaneously attains near optimal convergence rates for the gradient descent algorithm, for all ranges of strong convexity, smoothness, and Lipschitz parameters, without a-priory knowledge of these parameters.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 4; 2023: 13; 2024: 13; 2025: 14; 2026: 8.

55 papers.

Sort:

By area:

  1. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 129
  2. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  3. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  4. 2023FedExP: Speeding Up Federated Averaging via Extrapolationcited by 68
  5. 2022Making SGD Parameter-Freecited by 50
  6. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 50
  7. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  8. 2024Understanding and Improving Training-free Loss-based Diffusion Guidancecited by 31
  9. 2023Parameter-Agnostic Optimization under Relaxed Smoothnesscited by 29
  10. 2023Adaptive SGD with Polyak stepsize and Line-search: Robust Convergence and Variance Reductioncited by 23
  11. 2021Stochastic Mirror Descent: Convergence Analysis and Adaptive Variants via the Mirror Stochastic Polyak Stepsizecited by 22
  12. 2022Nest Your Adaptive Algorithm for Parameter-Agnostic Nonconvex Minimax Optimizationcited by 16
38 more
  1. 2023A Stochastic Proximal Polyak Step Sizecited by 15
  2. 2024Directional Smoothness and Gradient Methods: Convergence and Adaptivitycited by 15
  3. 2021Acceleration via Fractal Learning Rate Schedulescited by 13
  4. 2024Tuning-Free Stochastic Optimizationcited by 13
  5. 2024Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performancecited by 13
  6. 2024How Free is Parameter-Free Stochastic Optimization?cited by 12
  7. 2023The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networkscited by 11
  8. 2024The Price of Adaptivity in Stochastic Convex Optimizationcited by 11
  9. 2024Gradient-Variation Online Learning under Generalized Smoothnesscited by 11
  10. 2024An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizescited by 10
  11. 2022Amortized Proximal Optimizationcited by 9
  12. 2023Function Value Learning: Adaptive Learning Rates Based on the Polyak Stepsize and Function Splitting in ERMcited by 8
  13. 2022Adaptive Learning Rates for Faster Stochastic Gradient Methodscited by 7
  14. 2023Adaptive Federated Learning with Auto-Tuned Clientscited by 7
  15. 2023Adaptive, Doubly Optimal No-Regret Learning in Strongly Monotone and Exp-Concave Games with Gradient Feedbackcited by 6
  16. 2025New Perspectives on the Polyak Stepsize: Surrogate Functions and Negative Resultscited by 6
  17. 2021Towards Statistical and Computational Complexities of Polyak Step Size Gradient Descentcited by 5
  18. 2024Tuning-Free Bilevel Optimization: New Algorithms and Convergence Analysiscited by 5
  19. 2025Analysis of an Idealized Stochastic Polyak Method and its Application to Black-Box Model Distillationcited by 5
  20. 2026Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizerscited by 4
  21. 2024Parameter-free Clipped Gradient Descent Meets Polyakcited by 3
  22. 2024The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithmscited by 3
  23. 2025Aligned Multi Objective Optimizationcited by 3
  24. 2025Gradient Methods with Online Scaling Part I. Theoretical Foundationscited by 3
  25. 2025Personalized Convolutional Dictionary Learning of Physiological Time Seriescited by 2
  26. 2025Glocal Smoothness: Line search and adaptive step sizes can help in theory too!cited by 2
  27. 2025Sparse Polyak: an adaptive step size rule for high-dimensional M-estimationcited by 2
  28. 2025Taking the Road Less Scheduled with Adaptive Polyak Stepscited by 2
  29. 2025Safeguarded Stochastic Polyak Step Sizes for Non-smooth Optimization: Robust Performance Without Small (Sub)Gradientscited by 2
  30. 2025Stein Diffusion Guidance: Training-Free Posterior Correction for Sampling Beyond High-Density Regionscited by 1
  31. 2025Simple Optimizers for Convex Aligned Multi-Objective Optimizationcited by 1
  32. 2025Gradient Methods with Online Scaling Part II. Practical Aspectscited by 1
  33. 2026Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Schedulercited by 1
  34. 2024Corridor Geometry in Gradient-Based Optimizationcited by 0
  35. 2024MARINA-P: Superior Performance in Non-smooth Federated Optimization with Adaptive Stepsizescited by 0
  36. 2025Entropic Mirror Descent for Linear Systems: Polyak's Stepsize and Implicit Biascited by 0
  37. 2025Stochastic Adaptive Gradient Descent Without Descentcited by 0
  38. 2026Sven: Singular Value Descent as a Computationally Efficient Natural Gradient Methodcited by 0

50 most cited of 55 All 55 (JSON)