# L4: Practical loss-based stepsize adaptation for deep learning

arXiv: https://arxiv.org/abs/1802.05074 · HTML: https://mlsanity.com/paper/1802.05074/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Feb 2018.

## Built on by (most cited first, 26 of 26)

- [DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedule](https://mlsanity.com/paper/2302.12022/) (2023)
- [Making SGD Parameter-Free](https://mlsanity.com/paper/2205.02160/) (2022)
- [Neural Implicit Flow: a mesh-agnostic dimensionality reduction paradigm of spatio-temporal data](https://mlsanity.com/paper/2204.03216/) (2022)
- [Adaptive SGD with Polyak stepsize and Line-search: Robust Convergence and Variance Reduction](https://mlsanity.com/paper/2308.06058/) (2023)
- [Stochastic Mirror Descent: Convergence Analysis and Adaptive Variants via the Mirror Stochastic Polyak Stepsize](https://mlsanity.com/paper/2110.15412/) (2021)
- [Single-Call Stochastic Extragradient Methods for Structured Non-monotone Variational Inequalities: Improved Analysis under Weaker Conditions](https://mlsanity.com/paper/2302.14043/) (2023)
- [Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performance](https://mlsanity.com/paper/2406.04142/) (2024)
- [How to decay your learning rate](https://mlsanity.com/paper/2103.12682/) (2021)
- [An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes](https://mlsanity.com/paper/2407.04358/) (2024)
- [Amortized Proximal Optimization](https://mlsanity.com/paper/2203.00089/) (2022)
- [KOALA: A Kalman Optimization Algorithm with Loss Adaptivity](https://mlsanity.com/paper/2107.03331/) (2021)
- [QLABGrad: a Hyperparameter-Free and Convergence-Guaranteed Scheme for Deep Learning](https://mlsanity.com/paper/2302.00252/) (2023)
- [New Perspectives on the Polyak Stepsize: Surrogate Functions and Negative Results](https://mlsanity.com/paper/2505.20219/) (2025)
- [MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters](https://mlsanity.com/paper/2402.02342/) (2024)
- [A Stochastic Bundle Method for Interpolating Networks](https://mlsanity.com/paper/2201.12678/) (2022)
- [The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithms](https://mlsanity.com/paper/2405.19585/) (2024)
- [Empirically explaining SGD from a line search perspective](https://mlsanity.com/paper/2103.17132/) (2021)
- [Towards Simple and Provable Parameter-Free Adaptive Gradient Methods](https://mlsanity.com/paper/2412.19444/) (2024)
- [A Probabilistically Motivated Learning Rate Adaptation for Stochastic Optimization](https://mlsanity.com/paper/2102.10880/) (2021)
- [LRTuner: A Learning Rate Tuner for Deep Neural Networks](https://mlsanity.com/paper/2105.14526/) (2021)
- [Using a one dimensional parabolic model of the full-batch loss to estimate learning rates during training](https://mlsanity.com/paper/2108.13880/) (2021)
- [Learning to Accelerate by the Methods of Step-size Planning](https://mlsanity.com/paper/2204.01705/) (2022)
- [Locally Optimal Descent for Dynamic Stepsize Scheduling](https://mlsanity.com/paper/2311.13877/) (2023)
- [ZENITH: Automated Gradient Norm Informed Stochastic Optimization](https://mlsanity.com/paper/2601.15212/) (2026)
- [Scalable Hyperparameter-Divergent Ensemble Training with Automatic Learning Rate Exploration for Large Models](https://mlsanity.com/paper/2604.24708/) (2026)
- [Stochastic Heavy Ball with Polyak Step Size and Armijo Line Search: A General Convergence Analysis](https://mlsanity.com/paper/2609.36668/) (2026)

## Abstract (arXiv)

We propose a stepsize adaptation scheme for stochastic gradient descent. It operates directly with the loss function and rescales the gradient in order to make fixed predicted progress on the loss. We demonstrate its capabilities by conclusively improving the performance of Adam and Momentum optimizers. The enhanced optimizers with default hyperparameters consistently outperform their constant stepsize counterparts, even the best ones, without a measurable increase in computational cost. The performance is validated on multiple architectures including dense nets, CNNs, ResNets, and the recurrent Differential Neural Computer on classical datasets MNIST, fashion MNIST, CIFAR10 and others.