mlsanityData to 14 Oct 2026

A Continuous-Time View of Early Stopping for Least Squares

arXiv 1810.10082 ↗First posted 23 Oct 2018stat.ML cs.LGcited by 69

Not on the map. Why?

Abstract (arXiv)

We study the statistical properties of the iterates generated by gradient descent, applied to the fundamental problem of least squares regression. We take a continuous-time view, i.e., consider infinitesimal step sizes in gradient descent, in which case the iterates form a trajectory called gradient flow. Our primary focus is to compare the risk of gradient flow to that of ridge regression. Under the calibration t=1/λ---where t is the time parameter in gradient flow, and λ the tuning parameter in ridge regression---we prove that the risk of gradient flow is no less than 1.69 times that of ridge, along the entire path (for all t ≥ 0). This holds in finite samples with very weak assumptions on the data model (in particular, with no assumptions on the features X). We prove that the same relative risk bound holds for prediction risk, in an average sense over the underlying signal β0. Finally, we examine limiting risk expressions (under standard Marchenko-Pastur asymptotics), and give supporting numerical experiments.

Key facts. Cited by 69 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 15; 2023: 8; 2024: 13; 2025: 8; 2026: 14.

69 papers.

Sort:

By area:

  1. 2021Explaining Neural Scaling Lawscited by 363
  2. 2022If Influence Functions are the Answer, Then What is the Question?cited by 144
  3. 2024Model Collapse Demystified: The Case of Regressioncited by 82
  4. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  5. 2024Scaling and renormalization in high-dimensional regressioncited by 50
  6. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  7. 2023Continual Learning in Linear Classification on Separable Datacited by 34
  8. 2024A Statistical Theory of Regularization-Based Continual Learningcited by 33
  9. 2021The Benefits of Implicit Regularization from SGD in Least Squares Problemscited by 32
  10. 2022On Implicit Bias in Overparameterized Bilevel Optimizationcited by 30
  11. 2023Generalization in Kernel Regression Under Realistic Assumptionscited by 27
  12. 2021The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networkscited by 26
38 more
  1. 2021Multi-scale Feature Learning Dynamics: Insights for Double Descentcited by 24
  2. 2022Freeze then Train: Towards Provable Representation Learning under Spurious Correlations and Feature Noisecited by 22
  3. 2021RATT: Leveraging Unlabeled Data to Guarantee Generalizationcited by 20
  4. 2021Interpolation can hurt robust generalization even when there is no noisecited by 15
  5. 2022On the Implicit Bias Towards Minimal Depth of Deep Neural Networkscited by 12
  6. 2022Sobolev Acceleration and Statistical Optimality for Learning Elliptic Equations via Gradient Descentcited by 11
  7. 2024Towards Robust Out-of-Distribution Generalization Bounds via Sharpnesscited by 11
  8. 2024On Regularization via Early Stopping for Least Squares Regressioncited by 11
  9. 2023Generalized equivalences between subsampling and ridge regularizationcited by 10
  10. 2022Regularization-wise double descent: Why it occurs and how to eliminate itcited by 9
  11. 2025Benefits of Early Stopping in Gradient Descent for Overparameterized Logistic Regressioncited by 9
  12. 2022Risk Bounds of Multi-Pass SGD for Least Squares in the Interpolation Regimecited by 8
  13. 2025Optimal Rates in Continual Linear Regression via Increasing Regularizationcited by 8
  14. 2022Overparameterized random feature regression with nearly orthogonal datacited by 7
  15. 2021Nonparametric Regression with Shallow Overparameterized Neural Networks Trained by GD with Early Stoppingcited by 5
  16. 2021Comparing Classes of Estimators: When does Gradient Descent Beat Ridge Regression in Linear Models?cited by 5
  17. 2022On Optimal Early Stopping: Over-informative versus Under-informative Parametrizationcited by 4
  18. 2023Robust Linear Regression: Gradient-descent, Early-stopping, and Beyondcited by 4
  19. 2024Failures and Successes of Cross-Validation for Early-Stopped Gradient Descentcited by 4
  20. 2025Learning Curves of Stochastic Gradient Descent in Kernel Regressioncited by 4
  21. 2022Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regressioncited by 3
  22. 2024Improving Implicit Regularization of SGD with Preconditioning for Least Square Problemscited by 3
  23. 2025On the Effectiveness of the z-Transform Method in Quadratic Optimizationcited by 3
  24. 2026Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenariocited by 3
  25. 2021Non-negative matrix factorization algorithms generally improve topic model fitscited by 2
  26. 2022High-dimensional Asymptotics of Langevin Dynamics in Spiked Matrix Modelscited by 2
  27. 2023Fast Robust Kernel Regression through Sign Gradient Descent with Early Stoppingcited by 2
  28. 2025Comparing regularisation paths of (conjugate) gradient estimators in ridge regressioncited by 2
  29. 2025Basic Inequalities for First-Order Optimization with Applications to Statistical Risk Analysiscited by 2
  30. 2022Resonance in Weight Space: Covariate Shift Can Drive Divergence of SGD with Momentumcited by 1
  31. 2023Flat Minima in Linear Estimation and an Extended Gauss Markov Theoremcited by 1
  32. 2024Thermodynamic limit in learning period threecited by 1
  33. 2024Better Locally Private Sparse Estimation Given Multiple Samples Per Usercited by 1
  34. 2024Implicit Regularization Paths of Weighted Neural Representationscited by 1
  35. 2026Hard labels sampled from sparse targets mislead rotation invariant algorithmscited by 1
  36. 2026Estimating Implicit Regularization in Deep Learningcited by 1
  37. 2026Double Descent and Malign Overfitting in Diffusion Modelscited by 1
  38. 2021AgFlow: Fast Model Selection of Penalized PCA via Implicit Regularization Effects of Gradient Flowcited by 0

50 most cited of 69 All 69 (JSON)