# A Brief Prehistory of Double Descent

arXiv: https://arxiv.org/abs/2004.04328 · HTML: https://mlsanity.com/paper/2004.04328/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2020.

## Built on by (most cited first, 30 of 44)

- [Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks](https://mlsanity.com/paper/2006.13198/) (2020)
- [Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation](https://mlsanity.com/paper/2105.14368/) (2021)
- [The Shape of Learning Curves: a Review](https://mlsanity.com/paper/2103.10948/) (2021)
- [High-dimensional analysis of double descent for linear regression with random projections](https://mlsanity.com/paper/2303.01372/) (2023)
- [Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositions](https://mlsanity.com/paper/2202.01331/) (2022)
- [Sparse Double Descent: Where Network Pruning Aggravates Overfitting](https://mlsanity.com/paper/2206.08684/) (2022)
- [When and how epochwise double descent happens](https://mlsanity.com/paper/2108.12006/) (2021)
- [Phenomenology of Double Descent in Finite-Width Neural Networks](https://mlsanity.com/paper/2203.07337/) (2022)
- [The Interpolating Information Criterion for Overparameterized Models](https://mlsanity.com/paper/2307.07785/) (2023)
- [Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes](https://mlsanity.com/paper/2210.07612/) (2022)
- [Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention](https://mlsanity.com/paper/2605.29548/) (2026)
- [Fundamental tradeoffs between memorization and robustness in random features and neural tangent regimes](https://mlsanity.com/paper/2106.02630/) (2021)
- [Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond](https://mlsanity.com/paper/2411.00247/) (2024)
- [On the Role of Optimization in Double Descent: A Least Squares Study](https://mlsanity.com/paper/2107.12685/) (2021)
- [Bias-variance decomposition of overparameterized regression with random linear features](https://mlsanity.com/paper/2203.05443/) (2022)
- [Unraveling the Enigma of Double Descent: An In-depth Analysis through the Lens of Learned Feature Space](https://mlsanity.com/paper/2310.13572/) (2023)
- [Classical Statistical (In-Sample) Intuitions Don't Generalize Well: A Note on Bias-Variance Tradeoffs, Overfitting and Moving from Fixed to Random Designs](https://mlsanity.com/paper/2409.18842/) (2024)
- [The Geometry of Over-parameterized Regression and Adversarial Perturbations](https://mlsanity.com/paper/2103.14108/) (2021)
- [Target alignment in truncated kernel ridge regression](https://mlsanity.com/paper/2206.14255/) (2022)
- [Efficient learning of nonlinear prediction models with time-series privileged information](https://mlsanity.com/paper/2209.07067/) (2022)
- [The Breakdown of Gaussian Universality in Classification of High-dimensional Linear Factor Mixtures](https://mlsanity.com/paper/2410.05609/) (2024)
- [Random Features Model with General Convex Regularization: A Fine Grained Analysis with Precise Asymptotic Learning Curves](https://mlsanity.com/paper/2204.02678/) (2022)
- [Data-Induced Interactions of Sparse Sensors Using Statistical Physics](https://mlsanity.com/paper/2307.11838/) (2023)
- [Minimum norm interpolation by perceptra: Explicit regularization and implicit bias](https://mlsanity.com/paper/2311.06138/) (2023)
- [Effect of Weight Quantization on Learning Models by Typical Case Analysis](https://mlsanity.com/paper/2401.17269/) (2024)
- [Not All Explanations for Deep Learning Phenomena Are Equally Valuable](https://mlsanity.com/paper/2506.23286/) (2025)
- [A Survey of Learning Curves with Bad Behavior: or How More Data Need Not Lead to Better Performance](https://mlsanity.com/paper/2211.14061/) (2022)
- [Quantifying lottery tickets under label noise: accuracy, calibration, and complexity](https://mlsanity.com/paper/2306.12190/) (2023)
- [A dynamic view of some anomalous phenomena in SGD](https://mlsanity.com/paper/2505.01751/) (2025)
- [Asymptotic Risk of Overparameterized Likelihood Models: Double Descent Theory for Deep Neural Networks](https://mlsanity.com/paper/2103.00500/) (2021)

## Abstract (arXiv)

In their thought-provoking paper [1], Belkin et al. illustrate and discuss the shape of risk curves in the context of modern high-complexity learners. Given a fixed training sample size n, such curves show the risk of a learner as a function of some (approximate) measure of its complexity N. With N the number of features, these curves are also referred to as feature curves. A salient observation in [1] is that these curves can display, what they call, double descent: with increasing N, the risk initially decreases, attains a minimum, and then increases until N equals n, where the training data is fitted perfectly. Increasing N even further, the risk decreases a second and final time, creating a peak at N=n. This twofold descent may come as a surprise, but as opposed to what [1] reports, it has not been overlooked historically. Our letter draws attention to some original, earlier findings, of interest to contemporary machine learning.