# A Continuous-Time View of Early Stopping for Least Squares

arXiv: https://arxiv.org/abs/1810.10082 · HTML: https://mlsanity.com/paper/1810.10082/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2018.

## Built on by (most cited first, 30 of 69)

- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [If Influence Functions are the Answer, Then What is the Question?](https://mlsanity.com/paper/2209.05364/) (2022)
- [Model Collapse Demystified: The Case of Regression](https://mlsanity.com/paper/2402.07712/) (2024)
- [Label-Imbalanced and Group-Sensitive Classification under Overparameterization](https://mlsanity.com/paper/2103.01550/) (2021)
- [Scaling and renormalization in high-dimensional regression](https://mlsanity.com/paper/2405.00592/) (2024)
- [Benign, Tempered, or Catastrophic: A Taxonomy of Overfitting](https://mlsanity.com/paper/2207.06569/) (2022)
- [Continual Learning in Linear Classification on Separable Data](https://mlsanity.com/paper/2306.03534/) (2023)
- [A Statistical Theory of Regularization-Based Continual Learning](https://mlsanity.com/paper/2406.06213/) (2024)
- [The Benefits of Implicit Regularization from SGD in Least Squares Problems](https://mlsanity.com/paper/2108.04552/) (2021)
- [On Implicit Bias in Overparameterized Bilevel Optimization](https://mlsanity.com/paper/2212.14032/) (2022)
- [Generalization in Kernel Regression Under Realistic Assumptions](https://mlsanity.com/paper/2312.15995/) (2023)
- [The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networks](https://mlsanity.com/paper/2110.03922/) (2021)
- [Multi-scale Feature Learning Dynamics: Insights for Double Descent](https://mlsanity.com/paper/2112.03215/) (2021)
- [Freeze then Train: Towards Provable Representation Learning under Spurious Correlations and Feature Noise](https://mlsanity.com/paper/2210.11075/) (2022)
- [RATT: Leveraging Unlabeled Data to Guarantee Generalization](https://mlsanity.com/paper/2105.00303/) (2021)
- [Interpolation can hurt robust generalization even when there is no noise](https://mlsanity.com/paper/2108.02883/) (2021)
- [On the Implicit Bias Towards Minimal Depth of Deep Neural Networks](https://mlsanity.com/paper/2202.09028/) (2022)
- [Sobolev Acceleration and Statistical Optimality for Learning Elliptic Equations via Gradient Descent](https://mlsanity.com/paper/2205.07331/) (2022)
- [Towards Robust Out-of-Distribution Generalization Bounds via Sharpness](https://mlsanity.com/paper/2403.06392/) (2024)
- [On Regularization via Early Stopping for Least Squares Regression](https://mlsanity.com/paper/2406.04425/) (2024)
- [Generalized equivalences between subsampling and ridge regularization](https://mlsanity.com/paper/2305.18496/) (2023)
- [Regularization-wise double descent: Why it occurs and how to eliminate it](https://mlsanity.com/paper/2206.01378/) (2022)
- [Benefits of Early Stopping in Gradient Descent for Overparameterized Logistic Regression](https://mlsanity.com/paper/2502.13283/) (2025)
- [Risk Bounds of Multi-Pass SGD for Least Squares in the Interpolation Regime](https://mlsanity.com/paper/2203.03159/) (2022)
- [Optimal Rates in Continual Linear Regression via Increasing Regularization](https://mlsanity.com/paper/2506.06501/) (2025)
- [Overparameterized random feature regression with nearly orthogonal data](https://mlsanity.com/paper/2211.06077/) (2022)
- [Nonparametric Regression with Shallow Overparameterized Neural Networks Trained by GD with Early Stopping](https://mlsanity.com/paper/2107.05341/) (2021)
- [Comparing Classes of Estimators: When does Gradient Descent Beat Ridge Regression in Linear Models?](https://mlsanity.com/paper/2108.11872/) (2021)
- [On Optimal Early Stopping: Over-informative versus Under-informative Parametrization](https://mlsanity.com/paper/2202.09885/) (2022)
- [Robust Linear Regression: Gradient-descent, Early-stopping, and Beyond](https://mlsanity.com/paper/2301.13486/) (2023)

## Abstract (arXiv)

We study the statistical properties of the iterates generated by gradient descent, applied to the fundamental problem of least squares regression. We take a continuous-time view, i.e., consider infinitesimal step sizes in gradient descent, in which case the iterates form a trajectory called gradient flow. Our primary focus is to compare the risk of gradient flow to that of ridge regression. Under the calibration t=1/λ---where t is the time parameter in gradient flow, and λ the tuning parameter in ridge regression---we prove that the risk of gradient flow is no less than 1.69 times that of ridge, along the entire path (for all t ≥ 0). This holds in finite samples with very weak assumptions on the data model (in particular, with no assumptions on the features X). We prove that the same relative risk bound holds for prediction risk, in an average sense over the underlying signal β0. Finally, we examine limiting risk expressions (under standard Marchenko-Pastur asymptotics), and give supporting numerical experiments.