mlsanityData to 7 Oct 2026

A Brief Prehistory of Double Descent

arXiv 2004.04328 ↗First posted 7 Apr 2020cs.LGcited by 44

Not on the map. Why?

Abstract (arXiv)

In their thought-provoking paper [1], Belkin et al. illustrate and discuss the shape of risk curves in the context of modern high-complexity learners. Given a fixed training sample size n, such curves show the risk of a learner as a function of some (approximate) measure of its complexity N. With N the number of features, these curves are also referred to as feature curves. A salient observation in [1] is that these curves can display, what they call, double descent: with increasing N, the risk initially decreases, attains a minimum, and then increases until N equals n, where the training data is fitted perfectly. Increasing N even further, the risk decreases a second and final time, creating a peak at N=n. This twofold descent may come as a surprise, but as opposed to what [1] reports, it has not been overlooked historically. Our letter draws attention to some original, earlier findings, of interest to contemporary machine learning.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 8; 2022: 11; 2023: 8; 2024: 7; 2025: 5; 2026: 4.

44 papers.

Sort:

By area:

  1. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  2. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  3. 2021The Shape of Learning Curves: a Reviewcited by 67
  4. 2023High-dimensional analysis of double descent for linear regression with random projectionscited by 47
  5. 2022Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositionscited by 31
  6. 2022Sparse Double Descent: Where Network Pruning Aggravates Overfittingcited by 26
  7. 2021When and how epochwise double descent happenscited by 20
  8. 2022Phenomenology of Double Descent in Finite-Width Neural Networkscited by 11
  9. 2023The Interpolating Information Criterion for Overparameterized Modelscited by 11
  10. 2022Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processescited by 10
  11. 2021Fundamental tradeoffs between memorization and robustness in random features and neural tangent regimescited by 9
  12. 2024Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyondcited by 9
32 more
  1. 2026Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentioncited by 8
  2. 2021On the Role of Optimization in Double Descent: A Least Squares Studycited by 7
  3. 2022Bias-variance decomposition of overparameterized regression with random linear featurescited by 7
  4. 2023Unraveling the Enigma of Double Descent: An In-depth Analysis through the Lens of Learned Feature Spacecited by 6
  5. 2024Classical Statistical (In-Sample) Intuitions Don't Generalize Well: A Note on Bias-Variance Tradeoffs, Overfitting and Moving from Fixed to Random Designscited by 5
  6. 2021The Geometry of Over-parameterized Regression and Adversarial Perturbationscited by 4
  7. 2022Target alignment in truncated kernel ridge regressioncited by 4
  8. 2022Efficient learning of nonlinear prediction models with time-series privileged informationcited by 4
  9. 2024The Breakdown of Gaussian Universality in Classification of High-dimensional Linear Factor Mixturescited by 4
  10. 2022Random Features Model with General Convex Regularization: A Fine Grained Analysis with Precise Asymptotic Learning Curvescited by 3
  11. 2023Data-Induced Interactions of Sparse Sensors Using Statistical Physicscited by 3
  12. 2023Minimum norm interpolation by perceptra: Explicit regularization and implicit biascited by 3
  13. 2024Effect of Weight Quantization on Learning Models by Typical Case Analysiscited by 3
  14. 2025Not All Explanations for Deep Learning Phenomena Are Equally Valuablecited by 3
  15. 2022A Survey of Learning Curves with Bad Behavior: or How More Data Need Not Lead to Better Performancecited by 2
  16. 2023Quantifying lottery tickets under label noise: accuracy, calibration, and complexitycited by 2
  17. 2025A dynamic view of some anomalous phenomena in SGDcited by 2
  18. 2021Asymptotic Risk of Overparameterized Likelihood Models: Double Descent Theory for Deep Neural Networkscited by 1
  19. 2022Do highly over-parameterized neural networks generalize since bad solutions are rare?cited by 1
  20. 2023No Data Augmentation? Alternative Regularizations for Effective Training on Small Datasetscited by 1
  21. 2024A Predictive Surrogate Model for Heat Transfer of an Impinging Jet on a Concave Surfacecited by 1
  22. 2025A Two-Phase Perspective on Deep Learning Dynamicscited by 1
  23. 2025LCDB 1.1: A Database Illustrating Learning Curves Are More Ill-Behaved Than Previously Thoughtcited by 1
  24. 2021Fitting Elephantscited by 0
  25. 2022Generalizing similarity in noisy setups: the DIBS phenomenoncited by 0
  26. 2023Analysis of Interpolating Regression Models and the Double Descent Phenomenoncited by 0
  27. 2024Class-wise Activation Unravelling the Engima of Deep Double Descentcited by 0
  28. 2024eGAD! double descent is explained by Generalized Aliasing Decompositioncited by 0
  29. 2025On The Presence of Double-Descent in Deep Reinforcement Learningcited by 0
  30. 2026Origins and mitigation of double descent in reduced order modelingcited by 0
  31. 2026Benign interpolation and Occam's razorcited by 0
  32. 2026The double descent and Runge phenomena in overparametrized polynomial interpolationcited by 0