mlsanityData to 14 Oct 2026

L4: Practical loss-based stepsize adaptation for deep learning

arXiv 1802.05074 ↗First posted 14 Feb 2018cs.LG stat.MLcited by 26

Not on the map. Why?

Abstract (arXiv)

We propose a stepsize adaptation scheme for stochastic gradient descent. It operates directly with the loss function and rescales the gradient in order to make fixed predicted progress on the loss. We demonstrate its capabilities by conclusively improving the performance of Adam and Momentum optimizers. The enhanced optimizers with default hyperparameters consistently outperform their constant stepsize counterparts, even the best ones, without a measurable increase in computational cost. The performance is validated on multiple architectures including dense nets, CNNs, ResNets, and the recurrent Differential Neural Computer on classical datasets MNIST, fashion MNIST, CIFAR10 and others.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 5; 2023: 5; 2024: 5; 2025: 1; 2026: 3.

26 papers.

Sort:

  1. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  2. 2022Making SGD Parameter-Freecited by 50
  3. 2022Neural Implicit Flow: a mesh-agnostic dimensionality reduction paradigm of spatio-temporal datacited by 37
  4. 2023Adaptive SGD with Polyak stepsize and Line-search: Robust Convergence and Variance Reductioncited by 23
  5. 2021Stochastic Mirror Descent: Convergence Analysis and Adaptive Variants via the Mirror Stochastic Polyak Stepsizecited by 22
  6. 2023Single-Call Stochastic Extragradient Methods for Structured Non-monotone Variational Inequalities: Improved Analysis under Weaker Conditionscited by 14
  7. 2024Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performancecited by 13
  8. 2021How to decay your learning ratecited by 11
  9. 2024An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizescited by 10
  10. 2022Amortized Proximal Optimizationcited by 9
  11. 2021KOALA: A Kalman Optimization Algorithm with Loss Adaptivitycited by 6
  12. 2023QLABGrad: a Hyperparameter-Free and Convergence-Guaranteed Scheme for Deep Learningcited by 6
14 more
  1. 2025New Perspectives on the Polyak Stepsize: Surrogate Functions and Negative Resultscited by 6
  2. 2024MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameterscited by 5
  3. 2022A Stochastic Bundle Method for Interpolating Networkscited by 3
  4. 2024The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithmscited by 3
  5. 2021Empirically explaining SGD from a line search perspectivecited by 2
  6. 2024Towards Simple and Provable Parameter-Free Adaptive Gradient Methodscited by 2
  7. 2021A Probabilistically Motivated Learning Rate Adaptation for Stochastic Optimizationcited by 0
  8. 2021LRTuner: A Learning Rate Tuner for Deep Neural Networkscited by 0
  9. 2021Using a one dimensional parabolic model of the full-batch loss to estimate learning rates during trainingcited by 0
  10. 2022Learning to Accelerate by the Methods of Step-size Planningcited by 0
  11. 2023Locally Optimal Descent for Dynamic Stepsize Schedulingcited by 0
  12. 2026ZENITH: Automated Gradient Norm Informed Stochastic Optimizationcited by 0
  13. 2026Scalable Hyperparameter-Divergent Ensemble Training with Automatic Learning Rate Exploration for Large Modelscited by 0
  14. 2026Stochastic Heavy Ball with Polyak Step Size and Armijo Line Search: A General Convergence Analysiscited by 0