mlsanityData to 7 Oct 2026

Two models of double descent for weak features

arXiv 1903.07571 ↗First posted 18 Mar 2019cs.LG stat.MLcited by 194

Not on the map. Why?

Abstract (arXiv)

The "double descent" risk curve was proposed to qualitatively describe the out-of-sample prediction accuracy of variably-parameterized machine learning models. This article provides a precise mathematical analysis for the shape of this curve in two simple data models with the least squares/least norm predictor. Specifically, it is shown that the risk peaks when the number of features p is close to the sample size n, but also that the risk decreases towards its minimum as p increases beyond n. This behavior is contrasted with that of "prescient" models that select features in an a priori optimal order.

Key facts. Cited by 194 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 49; 2022: 46; 2023: 35; 2024: 27; 2025: 19; 2026: 18.

194 papers.

Sort:

By area:

  1. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  2. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  3. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  4. 2022The CLRS Algorithmic Reasoning Benchmarkcited by 103
  5. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  6. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  7. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  8. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
  9. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  10. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  11. 2023Theory on Forgetting and Generalization of Continual Learningcited by 69
  12. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
38 more
  1. 2021A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learningcited by 54
  2. 2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 48
  3. 2023High-dimensional analysis of double descent for linear regression with random projectionscited by 47
  4. 2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 41
  5. 2022The Implicit Bias of Benign Overfittingcited by 38
  6. 2021Last Iterate Risk Bounds of SGD with Decaying Stepsize for Overparameterized Linear Regressioncited by 37
  7. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  8. 2023Deep networks for system identification: a Surveycited by 35
  9. 2021Early-stopped neural networks are consistentcited by 34
  10. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
  11. 2023Benign Overfitting for Two-layer ReLU Convolutional Neural Networkscited by 34
  12. 2021Towards an Understanding of Benign Overfitting in Neural Networkscited by 29
  13. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  14. 2022Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regimecited by 28
  15. 2022On the Impossible Safety of Large AI Modelscited by 27
  16. 2021Multi-scale Feature Learning Dynamics: Insights for Double Descentcited by 24
  17. 2022The good, the bad and the ugly sides of data augmentation: An implicit spectral regularization perspectivecited by 24
  18. 2023Precise Asymptotic Analysis of Deep Random Feature Modelscited by 22
  19. 2021The Three Stages of Learning Dynamics in High-Dimensional Kernel Methodscited by 21
  20. 2022Random matrix analysis of deep neural network weight matricescited by 21
  21. 2022Overparameterized Linear Regression under Adversarial Attackscited by 21
  22. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  23. 2021Simple, Fast, and Flexible Framework for Matrix Completion with Infinite Width Neural Networkscited by 17
  24. 2021Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature modelcited by 17
  25. 2021Optimistic Rates: A Unifying Theory for Interpolation Learning and Regularization in Linear Regressioncited by 17
  26. 2022A Blessing of Dimensionality in Membership Inference through Regularizationcited by 17
  27. 2023Regularization properties of adversarially-trained linear regressioncited by 17
  28. 2023From Tempered to Benign Overfitting in ReLU Neural Networkscited by 16
  29. 2024Provable Weak-to-Strong Generalization via Benign Overfittingcited by 16
  30. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  31. 2021An Exponential Improvement on the Memorization Capacity of Deep Threshold Networkscited by 14
  32. 2021Harmless interpolation in regression and classification with structured featurescited by 14
  33. 2022Parameters or Privacy: A Provable Tradeoff Between Overparameterization and Membership Inferencecited by 14
  34. 2022Memorize to Generalize: on the Necessity of Interpolation in High Dimensional Linear Regressioncited by 14
  35. 2021On the Generalization Power of Overfitted Two-Layer Neural Tangent Kernel Modelscited by 13
  36. 2022On the Inconsistency of Kernel Ridgeless Regression in Fixed Dimensionscited by 13
  37. 2022Homophily modulates double descent generalization in graph convolution networkscited by 13
  38. 2021A Representation Learning Perspective on the Importance of Train-Validation Splitting in Meta-Learningcited by 12

50 most cited of 194 All 194 (JSON)