mlsanityData to 7 Oct 2026

Benefits of Early Stopping in Gradient Descent for Overparameterized Logistic Regression

arXiv 2502.13283 ↗First posted 18 Feb 2025cs.LG stat.MLcited by 9

Not on the map. Why?

Abstract (arXiv)

In overparameterized logistic regression, gradient descent (GD) iterates diverge in norm while converging in direction to the maximum ℓ2-margin solution -- a phenomenon known as the implicit bias of GD. This work investigates additional regularization effects induced by early stopping in well-specified high-dimensional logistic regression. We first demonstrate that the excess logistic risk vanishes for early-stopped GD but diverges to infinity for GD iterates at convergence. This suggests that early-stopped GD is well-calibrated, whereas asymptotic GD is statistically inconsistent. Second, we show that to attain a small excess zero-one risk, polynomially many samples are sufficient for early-stopped GD, while exponentially many samples are necessary for any interpolating estimator, including asymptotic GD. This separation underscores the statistical benefits of early stopping in the overparameterized regime. Finally, we establish nonasymptotic bounds on the norm and angular differences between early-stopped GD and ℓ2-regularized empirical risk minimizer, thereby connecting the implicit regularization of GD with explicit ℓ2-regularization.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 36 of its 57 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

36 of 57 references.

References · 36

  1. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  2. 2017Exploring Generalization in Deep Learningcited by 595
  3. 2019Benign Overfitting in Linear Regressioncited by 544
  4. 2021Deep learning: a statistical viewpointcited by 239
  5. 2020Benign overfitting in ridge regressioncited by 152
  6. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
  7. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
  8. 2018Risk and parameter convergence of logistic regressioncited by 81
  9. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  10. 2018A modern maximum-likelihood theory for high-dimensional logistic regressioncited by 79
  11. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  12. 2018A Continuous-Time View of Early Stopping for Least Squarescited by 67
24 more
  1. 2020Gradient descent follows the regularization path for general lossescited by 57
  2. 2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 48
  3. 2018The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regressioncited by 45
  4. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  5. 2022The Implicit Bias of Benign Overfittingcited by 38
  6. 2021Last Iterate Risk Bounds of SGD with Decaying Stepsize for Overparameterized Linear Regressioncited by 37
  7. 2013Margins, Shrinkage, and Boostingcited by 35
  8. 2021Early-stopped neural networks are consistentcited by 34
  9. 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 34
  10. 2013Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regressioncited by 32
  11. 2020On the proliferation of support vectors in high dimensionscited by 32
  12. 2021The Benefits of Implicit Regularization from SGD in Least Squares Problemscited by 32
  13. 2018Finite-sample analysis of M-estimators using self-concordancecited by 30
  14. 2020Gradient Methods Never Overfit On Separable Datacited by 29
  15. 2019Beyond Least-Squares: Fast Rates for Regularized Empirical Risk Minimization through Self-Concordancecited by 28
  16. 2019The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regimecited by 27
  17. 2020Binary Classification of Gaussian Mixtures: Abundance of Support Vectors, Benign Overfitting and Regularizationcited by 27
  18. 2022The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shiftcited by 25
  19. 2022Stochastic linear optimization never overfits with quadratically-bounded losses on general datacited by 16
  20. 2023On the sample complexity of parameter estimation in logistic regression with normal designcited by 13
  21. 2024On Regularization via Early Stopping for Least Squares Regressioncited by 10
  22. 2022Risk Bounds of Multi-Pass SGD for Least Squares in the Interpolation Regimecited by 8
  23. 2023Tight Risk Bounds for Gradient Descent on Separable Datacited by 7
  24. 2024Finite-sample performance of the maximum likelihood estimator in logistic regressioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 3.

9 papers.

  1. 2025Large Stepsizes Accelerate Gradient Descent for Regularized Logistic Regressioncited by 7
  2. 2025Comparing regularisation paths of (conjugate) gradient estimators in ridge regressioncited by 2
  3. 2025Implicit bias produces neural scaling laws in learning curves, from perceptrons to deep networkscited by 2
  4. 2025Basic Inequalities for First-Order Optimization with Applications to Statistical Risk Analysiscited by 2
  5. 2026Too Sharp, Too Sure: When Calibration Follows Curvaturecited by 2
  6. 2025Early-stopping for Transformer model trainingcited by 1
  7. 2025The Interplay of Statistics and Noisy Optimization: Learning Linear Predictors with Random Data Weightscited by 0
  8. 2026Self-Regularized Learning Methodscited by 0
  9. 2026Hard labels sampled from sparse targets mislead rotation invariant algorithmscited by 0