# On Convergence-Diagnostic based Step Sizes for Stochastic Gradient Descent

arXiv: https://arxiv.org/abs/2007.00534 · HTML: https://mlsanity.com/paper/2007.00534/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jul 2020.

## Built on by (most cited first, 11 of 11)

- [SVRG Meets AdaGrad: Painless Variance Reduction](https://mlsanity.com/paper/2102.09645/) (2021)
- [Stochastic Reweighted Gradient Descent](https://mlsanity.com/paper/2103.12293/) (2021)
- [On the Convergence of Federated Learning Algorithms without Data Similarity](https://mlsanity.com/paper/2403.02347/) (2024)
- [Almost Bayesian: The Fractal Dynamics of Stochastic Gradient Descent](https://mlsanity.com/paper/2503.22478/) (2025)
- [AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descent](https://mlsanity.com/paper/2505.21651/) (2025)
- [Trust, but Verify: Peeling Low-Bit Transformer Networks for Training Monitoring](https://mlsanity.com/paper/2605.02853/) (2026)
- [Adaptive Learning Rate and Momentum for Training Deep Neural Networks](https://mlsanity.com/paper/2106.11548/) (2021)
- [Cost-Efficient Distributed Learning via Combinatorial Multi-Armed Bandits](https://mlsanity.com/paper/2202.08302/) (2022)
- [Fast and Straggler-Tolerant Distributed SGD with Reduced Computation Load](https://mlsanity.com/paper/2304.08589/) (2023)
- [Beyond Discretization: Learning the Optimal Solution Path](https://mlsanity.com/paper/2410.14885/) (2024)
- [Coupling-based Convergence Diagnostic and Stepsize Scheme for Stochastic Gradient Descent](https://mlsanity.com/paper/2412.11341/) (2024)

## Abstract (arXiv)

Constant step-size Stochastic Gradient Descent exhibits two phases: a transient phase during which iterates make fast progress towards the optimum, followed by a stationary phase during which iterates oscillate around the optimal point. In this paper, we show that efficiently detecting this transition and appropriately decreasing the step size can lead to fast convergence rates. We analyse the classical statistical test proposed by Pflug (1983), based on the inner product between consecutive stochastic gradients. Even in the simple case where the objective function is quadratic we show that this test cannot lead to an adequate convergence diagnostic. We then propose a novel and simple statistical procedure that accurately detects stationarity and we provide experimental results showing state-of-the-art performance on synthetic and real-world datasets.