# GRADSTOP: Early Stopping of Gradient Descent via Posterior Sampling

arXiv: https://arxiv.org/abs/2508.19028 · HTML: https://mlsanity.com/paper/2508.19028/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 60 references that are in the map.
- First posted on arXiv 26 Aug 2025.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [SELF: Learning to Filter Noisy Labels with Self-Ensembling](https://mlsanity.com/paper/1910.01842/) (2019)
- [Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks](https://mlsanity.com/paper/1903.11680/) (2019)
- [Understanding and Improving Early Stopping for Learning with Noisy Labels](https://mlsanity.com/paper/2106.15853/) (2021)
- [Early stopping and non-parametric regression: An optimal data-dependent stopping rule](https://mlsanity.com/paper/1306.3574/) (2013)
- [A Continuous-Time View of Early Stopping for Least Squares](https://mlsanity.com/paper/1810.10082/) (2018)
- [In Defense of the Unitary Scalarization for Deep Multi-Task Learning](https://mlsanity.com/paper/2201.04122/) (2022)
- [A Variational Analysis of Stochastic Gradient Algorithms](https://mlsanity.com/paper/1602.02666/) (2016)
- [Deep Networks Always Grok and Here is Why](https://mlsanity.com/paper/2402.15555/) (2024)
- [Stiffness: A New Perspective on Generalization in Neural Networks](https://mlsanity.com/paper/1901.09491/) (2019)
- [Early Stopping in Deep Networks: Double Descent and How to Eliminate it](https://mlsanity.com/paper/2007.10099/) (2020)
- [Understanding Why Neural Networks Generalize Well Through GSNR of Parameters](https://mlsanity.com/paper/2001.07384/) (2020)
- [Early Stopping without a Validation Set](https://mlsanity.com/paper/1703.09580/) (2017)
- [Early stopping for kernel boosting algorithms: A general analysis with localized complexities](https://mlsanity.com/paper/1707.01543/) (2017)
- [Early Stopping Against Label Noise Without Validation Data](https://mlsanity.com/paper/2502.07551/) (2025)
- [The Statistical Complexity of Early-Stopped Mirror Descent](https://mlsanity.com/paper/2002.00189/) (2020)
- [Channel-Wise Early Stopping without a Validation Set via NNK Polytope Interpolation](https://mlsanity.com/paper/2107.12972/) (2021)
- [Disparity Between Batches as a Signal for Early Stopping](https://mlsanity.com/paper/2107.06665/) (2021)
- [Don't Waste Your Time: Early Stopping Cross-Validation](https://mlsanity.com/paper/2405.03389/) (2024)
- [On Optimal Early Stopping: Over-informative versus Under-informative Parametrization](https://mlsanity.com/paper/2202.09885/) (2022)
- [Failures and Successes of Cross-Validation for Early-Stopped Gradient Descent](https://mlsanity.com/paper/2402.16793/) (2024)

## Abstract (arXiv)

Machine learning models are often learned by minimising a loss function on the training data using a gradient descent algorithm. These models often suffer from overfitting, leading to a decline in predictive performance on unseen data. A standard solution is early stopping using a hold-out validation set, which halts the minimisation when the validation loss stops decreasing. However, this hold-out set reduces the data available for training. This paper presents GRADSTOP, a novel stochastic early stopping method that only uses information in the gradients, which are produced by the gradient descent algorithm ``for free.'' Our main contributions are that we estimate the Bayesian posterior by the gradient information, define the early stopping problem as drawing sample from this posterior, and use the approximated posterior to obtain a stopping criterion. Our empirical evaluation shows that GRADSTOP achieves a small loss on test data and compares favourably to a validation-set-based stopping criterion. By leveraging the entire dataset for training, our method is particularly advantageous in data-limited settings, such as transfer learning. It can be incorporated as an optional feature in gradient descent libraries with only a small computational overhead. The source code is available at https://github.com/edahelsinki/gradstop.