# Stochastic gradient descent with random learning rate

arXiv: https://arxiv.org/abs/2003.06926 · HTML: https://mlsanity.com/paper/2003.06926/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Mar 2020.

## Built on by (most cited first, 4 of 4)

- [Regime-Switching Langevin Monte Carlo Algorithms](https://mlsanity.com/paper/2509.00941/) (2025)
- [Cyclic and Randomized Stepsizes Invoke Heavier Tails in SGD than Constant Stepsize](https://mlsanity.com/paper/2302.05516/) (2023)
- [A Comprehensive Study on Optimization Strategies for Gradient Descent In Deep Learning](https://mlsanity.com/paper/2101.02397/) (2021)
- [Effect of Random Learning Rate: Theoretical Analysis of SGD Dynamics in Non-Convex Optimization via Stationary Distribution](https://mlsanity.com/paper/2406.16032/) (2024)

## Abstract (arXiv)

We propose to optimize neural networks with a uniformly-distributed random learning rate. The associated stochastic gradient descent algorithm can be approximated by continuous stochastic equations and analyzed within the Fokker-Planck formalism. In the small learning rate regime, the training process is characterized by an effective temperature which depends on the average learning rate, the mini-batch size and the momentum of the optimization algorithm. By comparing the random learning rate protocol with cyclic and constant protocols, we suggest that the random choice is generically the best strategy in the small learning rate regime, yielding better regularization without extra computational cost. We provide supporting evidence through experiments on both shallow, fully-connected and deep, convolutional neural networks for image classification on the MNIST and CIFAR10 datasets.