mlsanityData to 14 Oct 2026

Least Squares Regression Can Exhibit Under-Parameterized Double Descent

arXiv 2305.14689 ↗First posted 24 May 2023stat.ML cs.LG math.STcited by 4

Not on the map. Why?

Abstract (arXiv)

The relationship between the number of training data points, the number of parameters, and the generalization capabilities of models has been widely studied. Previous work has shown that double descent can occur in the over-parameterized regime and that the standard bias-variance trade-off holds in the under-parameterized regime. These works provide multiple reasons for the existence of the peak. We postulate that the location of the peak depends on the technical properties of both the spectrum as well as the eigenvectors of the sample covariance. We present two simple examples that provably exhibit double descent in the under-parameterized regime and do not seem to occur for reasons provided in prior work.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 36 of its 56 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

36 of 56 references.

References · 36

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  3. 2019Benign Overfitting in Linear Regressioncited by 549
  4. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  5. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  6. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  7. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  8. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  9. 2019Two models of double descent for weak featurescited by 195
  10. 2019Linearized two-layers neural networks in high dimensioncited by 155
  11. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  12. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 148
24 more
  1. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 127
  2. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  3. 2019Harmless interpolation of noisy data in regressioncited by 121
  4. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  5. 2020Optimal Regularization Can Mitigate Double Descentcited by 109
  6. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  7. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 98
  8. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  9. 2018A modern maximum-likelihood theory for high-dimensional logistic regressioncited by 81
  10. 2019A Model of Double Descent for High-dimensional Binary Linear Classificationcited by 81
  11. 2020Triple descent and the two kinds of overfitting: Where & why do they appear?cited by 65
  12. 2020The role of regularization in classification of high-dimensional noisy Gaussian mixturecited by 63
  13. 2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 54
  14. 2018Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularizationcited by 53
  15. 2022Dimension free ridge regressioncited by 51
  16. 2020Multiple Descent: Design Your Own Generalization Curvecited by 45
  17. 2019Exact expressions for double descent and implicit regularization via surrogate random designcited by 35
  18. 2020Analytic Study of Double Descent in Binary Classification: The Impact of Losscited by 28
  19. 2019Eigenvalue distribution of nonlinear models of random matricescited by 27
  20. 2021Covariate Shift in High-Dimensional Random Feature Regressioncited by 24
  21. 2022Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regressioncited by 22
  22. 2021On the Inherent Regularization Effects of Noise Injection During Trainingcited by 18
  23. 2020Dimensionality reduction, regularization, and generalization in overparameterized regressionscited by 13
  24. 2022Regularization-wise double descent: Why it occurs and how to eliminate itcited by 9

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024Near-Interpolators: Rapid Norm Growth and the Trade-Off between Interpolation and Generalizationcited by 8
  2. 2025Low Rank Gradients and Where to Find Themcited by 4
  3. 2024Generalization for Least Squares Regression With Simple Spiked Covariancescited by 2
  4. 2025Matricial Free Energy as a Gaussianizing Regularizer: Enhancing Autoencoders for Gaussian Code Generationcited by 0