# A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning

arXiv: https://arxiv.org/abs/2310.18988 · HTML: https://mlsanity.com/paper/2310.18988/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 77 references that are in the map.
- First posted on arXiv 29 Oct 2023.

## Builds on (most cited first, 12 of 12)

- [In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning](https://mlsanity.com/paper/1412.6614/) (2014)
- [Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning](https://mlsanity.com/paper/1811.12808/) (2018)
- [A jamming transition from under- to over-parametrization affects loss landscape and generalization](https://mlsanity.com/paper/1810.09665/) (2018)
- [A Modern Take on the Bias-Variance Tradeoff in Neural Networks](https://mlsanity.com/paper/1810.08591/) (2018)
- [A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning](https://mlsanity.com/paper/2109.02355/) (2021)
- [Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited](https://mlsanity.com/paper/2003.02139/) (2020)
- [Double descent in the condition number](https://mlsanity.com/paper/1912.06190/) (2019)
- [On the Bias-Variance Tradeoff: Textbooks Need an Update](https://mlsanity.com/paper/1912.08286/) (2019)
- [Revisiting minimum description length complexity in overparameterized models](https://mlsanity.com/paper/2006.10189/) (2020)
- [Double Descent and Other Interpolation Phenomena in GANs](https://mlsanity.com/paper/2106.04003/) (2021)
- [There is no Double-Descent in Random Forests](https://mlsanity.com/paper/2111.04409/) (2021)
- [On the Approximation Properties of Random ReLU Features](https://mlsanity.com/paper/1810.04374/) (2018)

## Built on by (most cited first, 30 of 33)

- [How much do language models memorize?](https://mlsanity.com/paper/2505.24832/) (2025)
- [Deep Learning is Not So Mysterious or Different](https://mlsanity.com/paper/2503.02113/) (2025)
- [How Linguistics Learned to Stop Worrying and Love the Language Models](https://mlsanity.com/paper/2501.17047/) (2025)
- [Why do Random Forests Work? Understanding Tree Ensembles as Self-Regularizing Adaptive Smoothers](https://mlsanity.com/paper/2402.01502/) (2024)
- [Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond](https://mlsanity.com/paper/2411.00247/) (2024)
- [Double descent in quantum kernel methods](https://mlsanity.com/paper/2501.10077/) (2025)
- [Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representations](https://mlsanity.com/paper/2406.09366/) (2024)
- [Large Language Models and Emergence: A Complex Systems Perspective](https://mlsanity.com/paper/2506.11135/) (2025)
- [Classical Statistical (In-Sample) Intuitions Don't Generalize Well: A Note on Bias-Variance Tradeoffs, Overfitting and Moving from Fixed to Random Designs](https://mlsanity.com/paper/2409.18842/) (2024)
- [Towards Understanding Epoch-wise Double descent in Two-layer Linear Neural Networks](https://mlsanity.com/paper/2407.09845/) (2024)
- [Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarks](https://mlsanity.com/paper/2502.18339/) (2025)
- [Not All Explanations for Deep Learning Phenomena Are Equally Valuable](https://mlsanity.com/paper/2506.23286/) (2025)
- [Revisiting Optimism and Model Complexity in the Wake of Overparameterized Machine Learning](https://mlsanity.com/paper/2410.01259/) (2024)
- [A Note on Generalization in Variational Autoencoders: How Effective Is Synthetic Data & Overparameterization?](https://mlsanity.com/paper/2310.19653/) (2023)
- [Changing the Kernel During Training Leads to Double Descent in Kernel Regression](https://mlsanity.com/paper/2311.01762/) (2023)
- [ESS-ReduNet: Enhancing Subspace Separability of ReduNet via Dynamic Expansion with Bayesian Inference](https://mlsanity.com/paper/2411.17961/) (2024)
- [Manipulating Sparse Double Descent](https://mlsanity.com/paper/2401.10686/) (2024)
- [Consistency for Large Neural Networks: Regression and Classification](https://mlsanity.com/paper/2409.14123/) (2024)
- [Is Supervised Learning Really That Different from Unsupervised?](https://mlsanity.com/paper/2505.11006/) (2025)
- [Impact of Bottleneck Layers and Skip Connections on the Generalization of Linear Denoising Autoencoders](https://mlsanity.com/paper/2505.24668/) (2025)
- [Evaluating Double Descent in Machine Learning: Insights from Tree-Based Models Applied to a Genomic Prediction Task](https://mlsanity.com/paper/2509.25216/) (2025)
- [Language models as tools for investigating the distinction between possible and impossible natural languages](https://mlsanity.com/paper/2512.09394/) (2025)
- [Large language models have learned to use language](https://mlsanity.com/paper/2512.12447/) (2025)
- [Large language models are not about natural language](https://mlsanity.com/paper/2512.13441/) (2025)
- [Linguists should learn to love speech-based deep learning models](https://mlsanity.com/paper/2512.14506/) (2025)
- [Are Language Models Models?](https://mlsanity.com/paper/2601.10421/) (2026)
- [Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilities](https://mlsanity.com/paper/2604.09466/) (2026)
- [You Can't Fight in Here! This is BBS!](https://mlsanity.com/paper/2604.09501/) (2026)
- [Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appear](https://mlsanity.com/paper/2605.10061/) (2026)
- [Minimax Rates and Spectral Distillation for Tree Ensembles](https://mlsanity.com/paper/2605.11841/) (2026)

## Abstract (arXiv)

Conventional statistical wisdom established a well-understood relationship between model complexity and prediction error, typically presented as a U-shaped curve reflecting a transition between under- and overfitting regimes. However, motivated by the success of overparametrized neural networks, recent influential work has suggested this theory to be generally incomplete, introducing an additional regime that exhibits a second descent in test error as the parameter count p grows past sample size n - a phenomenon dubbed double descent. While most attention has naturally been given to the deep-learning setting, double descent was shown to emerge more generally across non-neural models: known cases include linear regression, trees, and boosting. In this work, we take a closer look at evidence surrounding these more classical statistical machine learning methods and challenge the claim that observed cases of double descent truly extend the limits of a traditional U-shaped complexity-generalization curve therein. We show that once careful consideration is given to what is being plotted on the x-axes of their double descent plots, it becomes apparent that there are implicitly multiple complexity axes along which the parameter count grows. We demonstrate that the second descent appears exactly (and only) when and where the transition between these underlying axes occurs, and that its location is thus not inherently tied to the interpolation threshold p=n. We then gain further insight by adopting a classical nonparametric statistics perspective. We interpret the investigated methods as smoothers and propose a generalized measure for the effective number of parameters they use on unseen examples, using which we find that their apparent double descent curves indeed fold back into more traditional convex shapes - providing a resolution to tensions between double descent and statistical intuition.