mlsanityData to 7 Oct 2026

On Regularization via Early Stopping for Least Squares Regression

arXiv 2406.04425 ↗First posted 6 Jun 2024cs.LG math.OC math.STcited by 10

Not on the map. Why?

Abstract (arXiv)

A fundamental problem in machine learning is understanding the effect of early stopping on the parameters obtained and the generalization capabilities of the model. Even for linear models, the effect is not fully understood for arbitrary learning rates and data. In this paper, we analyze the dynamics of discrete full batch gradient descent for linear regression. With minimal distributional assumptions, we characterize the trajectory of the parameters and the expected excess risk. Using this characterization, we show that when training with any learning rate schedule and finite time horizon, the early stopped solution is equivalent to the minimum norm solution for a generalized ridge regression problem. We also prove that early stopping is beneficial for generic data with arbitrary spectrum and for a wide variety of learning rate schedules. We provide an estimate for the optimal stopping time and empirically demonstrate the accuracy of our estimate.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 35 of its 54 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

35 of 54 references.

References · 35

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2019Benign Overfitting in Linear Regressioncited by 544
  6. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  7. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  8. 2019SGD: General Analysis and Improved Ratescited by 206
  9. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  10. 2020Optimal Regularization Can Mitigate Double Descentcited by 107
  11. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  12. 2013Early stopping and non-parametric regression: An optimal data-dependent stopping rulecited by 86
23 more
  1. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  2. 2018A Continuous-Time View of Early Stopping for Least Squarescited by 67
  3. 2020The Implicit Regularization of Stochastic Gradient Flow for Least Squarescited by 58
  4. 2020Implicit Regularization of Random Feature Modelscited by 54
  5. 2020Kernel Alignment Risk Estimator: Risk Prediction from Training Datacited by 54
  6. 2018Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularizationcited by 52
  7. 2022Dimension free ridge regressioncited by 50
  8. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 46
  9. 2022Homogenization of SGD in high-dimensions: Exact dynamics and generalization propertiescited by 26
  10. 2023Grokking in Linear Estimators -- A Solvable Model that Groks without Understandingcited by 25
  11. 2024Nonlinear spiked covariance matrices and signal propagation in deep neural networkscited by 25
  12. 2021Covariate Shift in High-Dimensional Random Feature Regressioncited by 24
  13. 2022Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensionscited by 23
  14. 2020Online Robust Regression via SGD on the l1 losscited by 20
  15. 2023High-dimensional Asymptotics of Denoising Autoencoderscited by 19
  16. 2021On the Inherent Regularization Effects of Noise Injection During Trainingcited by 18
  17. 2024Early alignment in two-layer networks training is a two-edged swordcited by 17
  18. 2023Double Descent and Overfitting under Noisy Inputs and Distribution Shift for Linear Denoiserscited by 13
  19. 2022Regularization-wise double descent: Why it occurs and how to eliminate itcited by 9
  20. 2024Near-Interpolators: Rapid Norm Growth and the Trade-Off between Interpolation and Generalizationcited by 7
  21. 2022On Optimal Early Stopping: Over-informative versus Under-informative Parametrizationcited by 4
  22. 2022Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regressioncited by 3
  23. 2024Stochastic gradient descent for streaming linear and rectified linear systems with adversarial corruptionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2025Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Timecited by 15
  2. 2025Benefits of Early Stopping in Gradient Descent for Overparameterized Logistic Regressioncited by 9
  3. 2025Models of Heavy-Tailed Mechanistic Universalitycited by 9
  4. 2025Design Considerations in Offline Preference-based RLcited by 4
  5. 2025Learning Curves of Stochastic Gradient Descent in Kernel Regressioncited by 3
  6. 2025Basic Inequalities for First-Order Optimization with Applications to Statistical Risk Analysiscited by 2
  7. 2026Conservative neural posterior estimation via distributionally robust trainingcited by 0
  8. 2026Ridge Regression from Poisson Resetting: A Renewal Perspective on Spectral Regularizationcited by 0
  9. 2026Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Negative-Shifted Gradient Descentcited by 0
  10. 2026A Data-dependent Early Stopping Rule using Rademacher Complexity with L1-normcited by 0