# Phenomenology of Double Descent in Finite-Width Neural Networks

arXiv: https://arxiv.org/abs/2203.07337 · HTML: https://mlsanity.com/paper/2203.07337/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 14 Mar 2022.

## Builds on (most cited first, 23 of 23)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [Influence Functions in Deep Learning Are Fragile](https://mlsanity.com/paper/2006.14651/) (2020)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond](https://mlsanity.com/paper/1611.07476/) (2016)
- [Harmless interpolation of noisy data in regression](https://mlsanity.com/paper/1903.09139/) (2019)
- [Optimal Regularization Can Mitigate Double Descent](https://mlsanity.com/paper/2003.01897/) (2020)
- [Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks](https://mlsanity.com/paper/2012.11654/) (2020)
- [Scaling description of generalization with number of parameters in deep learning](https://mlsanity.com/paper/1901.01608/) (2019)
- [Analytic Insights into Structure and Rank of Neural Network Hessian Maps](https://mlsanity.com/paper/2106.16225/) (2021)
- [A Brief Prehistory of Double Descent](https://mlsanity.com/paper/2004.04328/) (2020)
- [More Data Can Hurt for Linear Regression: Sample-wise Double Descent](https://mlsanity.com/paper/1912.07242/) (2019)
- [Scaling Down Deep Learning with MNIST-1D](https://mlsanity.com/paper/2011.14439/) (2020)
- [From Averaging to Acceleration, There is Only a Step-size](https://mlsanity.com/paper/1504.01577/) (2015)
- [Exponential convergence of testing error for stochastic gradient methods](https://mlsanity.com/paper/1712.04755/) (2017)
- [On the Role of Optimization in Double Descent: A Least Squares Study](https://mlsanity.com/paper/2107.12685/) (2021)

## Built on by (most cited first, 12 of 12)

- [Dataset Pruning: Reducing Training Data by Examining Generalization Influence](https://mlsanity.com/paper/2205.09329/) (2022)
- [Some Fundamental Aspects about Lipschitz Continuity of Neural Networks](https://mlsanity.com/paper/2302.10886/) (2023)
- [Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal States](https://mlsanity.com/paper/2211.10691/) (2022)
- [The Hessian perspective into the Nature of Convolutional Neural Networks](https://mlsanity.com/paper/2305.09088/) (2023)
- [Theoretical characterisation of the Gauss-Newton conditioning in Neural Networks](https://mlsanity.com/paper/2411.02139/) (2024)
- [Hallmarks of Optimization Trajectories in Neural Networks: Directional Exploration and Redundancy](https://mlsanity.com/paper/2403.07379/) (2024)
- [Understanding Overparametrization in Survival Models through Interpolation](https://mlsanity.com/paper/2512.12463/) (2025)
- [Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation](https://mlsanity.com/paper/2311.14120/) (2023)
- [Unraveling the Hessian: A Key to Smooth Convergence in Loss Function Landscapes](https://mlsanity.com/paper/2409.11995/) (2024)
- [Double descent for least-squares interpolation on contaminated data: A simulation study](https://mlsanity.com/paper/2605.21494/) (2026)
- [Improving Adversarial Robustness of Attribution via Implicit Regularization](https://mlsanity.com/paper/2605.29983/) (2026)
- [Cautious optimism for deep parameterized quantum circuits](https://mlsanity.com/paper/2607.21409/) (2026)

## Abstract (arXiv)

`Double descent' delineates the generalization behaviour of models depending on the regime they belong to: under- or over-parameterized. The current theoretical understanding behind the occurrence of this phenomenon is primarily based on linear and kernel regression models -- with informal parallels to neural networks via the Neural Tangent Kernel. Therefore such analyses do not adequately capture the mechanisms behind double descent in finite-width neural networks, as well as, disregard crucial components -- such as the choice of the loss function. We address these shortcomings by leveraging influence functions in order to derive suitable expressions of the population loss and its lower bound, while imposing minimal assumptions on the form of the parametric model. Our derived bounds bear an intimate connection with the spectrum of the Hessian at the optimum, and importantly, exhibit a double descent behaviour at the interpolation threshold. Building on our analysis, we further investigate how the loss function affects double descent -- and thus uncover interesting properties of neural networks and their Hessian spectra near the interpolation threshold.