# Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzle

arXiv: https://arxiv.org/abs/2303.14151 · HTML: https://mlsanity.com/paper/2303.14151/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 18 references that are in the map.
- First posted on arXiv 24 Mar 2023.

## Builds on (most cited first, 15 of 15)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks](https://mlsanity.com/paper/2002.02561/) (2020)
- [Optimal Regularization Can Mitigate Double Descent](https://mlsanity.com/paper/2003.01897/) (2020)
- [Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition](https://mlsanity.com/paper/2011.03321/) (2020)
- [Multiple Descent: Design Your Own Generalization Curve](https://mlsanity.com/paper/2008.01036/) (2020)
- [Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models](https://mlsanity.com/paper/2010.13933/) (2020)
- [Double descent in the condition number](https://mlsanity.com/paper/1912.06190/) (2019)
- [Bias-variance decomposition of overparameterized regression with random linear features](https://mlsanity.com/paper/2203.05443/) (2022)
- [The Geometry of Over-parameterized Regression and Adversarial Perturbations](https://mlsanity.com/paper/2103.14108/) (2021)

## Built on by (most cited first, 30 of 37)

- [RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs](https://mlsanity.com/paper/2404.08555/) (2024)
- [Deep Grokking: Would Deep Neural Networks Generalize Better?](https://mlsanity.com/paper/2405.19454/) (2024)
- [Grokking at the Edge of Linear Separability](https://mlsanity.com/paper/2410.04489/) (2024)
- [Amuro and Char: Analyzing the Relationship between Pre-Training and Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2408.06663/) (2024)
- [Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond](https://mlsanity.com/paper/2411.00247/) (2024)
- [Double descent in quantum kernel methods](https://mlsanity.com/paper/2501.10077/) (2025)
- [Post-Hoc Reversal: Are We Selecting Models Prematurely?](https://mlsanity.com/paper/2404.07815/) (2024)
- [Survival of the Fittest Representation: A Case Study with Modular Addition](https://mlsanity.com/paper/2405.17420/) (2024)
- [Large Language Models and Emergence: A Complex Systems Perspective](https://mlsanity.com/paper/2506.11135/) (2025)
- [High-dimensional analysis of ridge regression for non-identically distributed data with a variance profile](https://mlsanity.com/paper/2403.20200/) (2024)
- [Scaling Laws for Galaxy Images](https://mlsanity.com/paper/2404.02973/) (2024)
- [How more data can hurt: Instability and regularization in next-generation reservoir computing](https://mlsanity.com/paper/2407.08641/) (2024)
- [Swing-by Dynamics in Concept Learning and Compositional Generalization](https://mlsanity.com/paper/2410.08309/) (2024)
- [Unified Neural Network Scaling Laws and Scale-time Equivalence](https://mlsanity.com/paper/2409.05782/) (2024)
- [Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarks](https://mlsanity.com/paper/2502.18339/) (2025)
- [Neuroplasticity in Artificial Intelligence -- An Overview and Inspirations on Drop In & Out Learning](https://mlsanity.com/paper/2503.21419/) (2025)
- [Towards Understanding Task-agnostic Debiasing Through the Lenses of Intrinsic Bias and Forgetfulness](https://mlsanity.com/paper/2406.04146/) (2024)
- [Transformers as Implicit State Estimators: In-Context Learning in Dynamical Systems](https://mlsanity.com/paper/2410.16546/) (2024)
- [Aligning Generalisation Between Humans and Machines](https://mlsanity.com/paper/2411.15626/) (2024)
- [A dynamic view of some anomalous phenomena in SGD](https://mlsanity.com/paper/2505.01751/) (2025)
- [FlowVAT: Normalizing Flow Variational Inference with Affine-Invariant Tempering](https://mlsanity.com/paper/2505.10466/) (2025)
- [Quantifying the Effect of Test Set Contamination on Generative Evaluations](https://mlsanity.com/paper/2601.04301/) (2026)
- [Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation](https://mlsanity.com/paper/2311.14120/) (2023)
- [Linear Mode Connectivity under Data Shifts for Deep Ensembles of Image Classifiers](https://mlsanity.com/paper/2511.04514/) (2025)
- [Unveiling Scaling Behaviors in Molecular Language Models: Effects of Model Size, Data, and Representation](https://mlsanity.com/paper/2601.22757/) (2026)
- [Understanding the Role of Optimization in Double Descent](https://mlsanity.com/paper/2312.03951/) (2023)
- [eGAD! double descent is explained by Generalized Aliasing Decomposition](https://mlsanity.com/paper/2408.08294/) (2024)
- [Consistency for Large Neural Networks: Regression and Classification](https://mlsanity.com/paper/2409.14123/) (2024)
- [Path Regularization: A Near-Complete and Optimal Nonasymptotic Generalization Theory for Multilayer Neural Networks and Double Descent Phenomenon](https://mlsanity.com/paper/2503.02129/) (2025)
- [Approach to Finding a Robust Deep Learning Model](https://mlsanity.com/paper/2505.17254/) (2025)

## Abstract (arXiv)

Double descent is a surprising phenomenon in machine learning, in which as the number of model parameters grows relative to the number of data, test error drops as models grow ever larger into the highly overparameterized (data undersampled) regime. This drop in test error flies against classical learning theory on overfitting and has arguably underpinned the success of large models in machine learning. This non-monotonic behavior of test loss depends on the number of data, the dimensionality of the data and the number of model parameters. Here, we briefly describe double descent, then provide an explanation of why double descent occurs in an informal and approachable manner, requiring only familiarity with linear algebra and introductory probability. We provide visual intuition using polynomial regression, then mathematically analyze double descent with ordinary linear regression and identify three interpretable factors that, when simultaneously all present, together create double descent. We demonstrate that double descent occurs on real data when using ordinary linear regression, then demonstrate that double descent does not occur when any of the three factors are ablated. We use this understanding to shed light on recent observations in nonlinear models concerning superposition and double descent. Code is publicly available.