mlsanityData to 14 Oct 2026

On Optimal Early Stopping: Over-informative versus Under-informative Parametrization

arXiv 2202.09885 ↗First posted 20 Feb 2022cs.LG stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

Early stopping is a simple and widely used method to prevent over-training neural networks. We develop theoretical results to reveal the relationship between the optimal early stopping time and model dimension as well as sample size of the dataset for certain linear models. Our results demonstrate two very different behaviors when the model dimension exceeds the number of features versus the opposite scenario. While most previous works on linear models focus on the latter setting, we observe that the dimension of the model often exceeds the number of features arising from data in common deep learning tasks and propose a model to study this setting. We demonstrate experimentally that our theoretical results on optimal early stopping time corresponds to the training process of deep neural networks.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 29 of its 41 references that are in the map. First posted on arXiv 20 Feb 2022.

Builds on

29 of 41 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  5. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  6. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  7. 2019Benign Overfitting in Linear Regressioncited by 549
  8. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  9. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  10. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
  11. 2019Two models of double descent for weak featurescited by 195
  12. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
17 more
  1. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  2. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 159
  3. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 148
  4. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  5. 2019Harmless interpolation of noisy data in regressioncited by 121
  6. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 119
  7. 2020Optimal Regularization Can Mitigate Double Descentcited by 109
  8. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  9. 2019Implicit Regularization for Optimal Sparse Recoverycited by 80
  10. 2018A Continuous-Time View of Early Stopping for Least Squarescited by 69
  11. 2020The Implicit Regularization of Stochastic Gradient Flow for Least Squarescited by 59
  12. 2018Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularizationcited by 53
  13. 2020Multiple Descent: Design Your Own Generalization Curvecited by 45
  14. 2019Exact expressions for double descent and implicit regularization via surrogate random designcited by 35
  15. 2019Understanding overfitting peaks in generalization error: Analytical risk curves for l2 and l1 penalized interpolationcited by 23
  16. 2020The Statistical Complexity of Early-Stopped Mirror Descentcited by 19
  17. 2019WONDER: Weighted one-shot distributed ridge regression in high dimensionscited by 14

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 1; 2025: 1.

4 papers.

  1. 2024On Regularization via Early Stopping for Least Squares Regressioncited by 11
  2. 2022E Pluribus Unum Interpretable Convolutional Neural Networkscited by 3
  3. 2023On Emergence of Clean-Priority Learning in Early Stopped Neural Networkscited by 3
  4. 2025GRADSTOP: Early Stopping of Gradient Descent via Posterior Samplingcited by 0