# Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition

arXiv: https://arxiv.org/abs/2011.03321 · HTML: https://mlsanity.com/paper/2011.03321/

## Key facts

- Cited by 75 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 4 Nov 2020.

## Built on by (most cited first, 30 of 75)

- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [A Dynamical Model of Neural Scaling Laws](https://mlsanity.com/paper/2402.01092/) (2024)
- [Deep Ensembles Work, But Are They Necessary?](https://mlsanity.com/paper/2202.06985/) (2022)
- [Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective](https://mlsanity.com/paper/2203.08124/) (2022)
- [A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks](https://mlsanity.com/paper/2310.07891/) (2023)
- [Scaling and renormalization in high-dimensional regression](https://mlsanity.com/paper/2405.00592/) (2024)
- [Feature-Learning Networks Are Consistent Across Widths At Realistic Scales](https://mlsanity.com/paper/2305.18411/) (2023)
- [Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzle](https://mlsanity.com/paper/2303.14151/) (2023)
- [Deterministic equivalent and error universality of deep random features learning](https://mlsanity.com/paper/2302.00401/) (2023)
- [The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimes](https://mlsanity.com/paper/2212.12147/) (2022)
- [Contrasting random and learned features in deep Bayesian linear regression](https://mlsanity.com/paper/2203.00573/) (2022)
- [Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions](https://mlsanity.com/paper/2206.07252/) (2022)
- [Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression](https://mlsanity.com/paper/2205.14846/) (2022)
- [Uncertainty Estimates of Predictions via a General Bias-Variance Decomposition](https://mlsanity.com/paper/2210.12256/) (2022)
- [Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics for Convex Losses in High-Dimension](https://mlsanity.com/paper/2201.13383/) (2022)
- [Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer](https://mlsanity.com/paper/2502.02531/) (2025)
- [Pathologies of Predictive Diversity in Deep Ensembles](https://mlsanity.com/paper/2302.00704/) (2023)
- [AEDFL: Efficient Asynchronous Decentralized Federated Learning with Heterogeneous Devices](https://mlsanity.com/paper/2312.10935/) (2023)
- [Harmless interpolation in regression and classification with structured features](https://mlsanity.com/paper/2111.05198/) (2021)
- [Exact Gap between Generalization Error and Uniform Convergence in Random Feature Models](https://mlsanity.com/paper/2103.04554/) (2021)
- [Learning curves for deep structured Gaussian feature models](https://mlsanity.com/paper/2303.00564/) (2023)
- [Demystifying Disagreement-on-the-Line in High Dimensions](https://mlsanity.com/paper/2301.13371/) (2023)
- [Subsample Ridge Ensembles: Equivalences and Generalized Cross-Validation](https://mlsanity.com/paper/2304.13016/) (2023)
- [Understanding Generalization in Adversarial Training via the Bias-Variance Decomposition](https://mlsanity.com/paper/2103.09947/) (2021)
- [Understanding the bias-variance tradeoff of Bregman divergences](https://mlsanity.com/paper/2202.04167/) (2022)
- [Generalization Through The Lens Of Leave-One-Out Error](https://mlsanity.com/paper/2203.03443/) (2022)
- [Multiple Descent in the Multiple Random Feature Model](https://mlsanity.com/paper/2208.09897/) (2022)
- [Generalized equivalences between subsampling and ridge regularization](https://mlsanity.com/paper/2305.18496/) (2023)
- [Redundant representations help generalization in wide neural networks](https://mlsanity.com/paper/2106.03485/) (2021)
- [Knowledge Sheaves: A Sheaf-Theoretic Framework for Knowledge Graph Embedding](https://mlsanity.com/paper/2110.03789/) (2021)

## Abstract (arXiv)

Classical learning theory suggests that the optimal generalization performance of a machine learning model should occur at an intermediate model complexity, with simpler models exhibiting high bias and more complex models exhibiting high variance of the predictive function. However, such a simple trade-off does not adequately describe deep learning models that simultaneously attain low bias and variance in the heavily overparameterized regime. A primary obstacle in explaining this behavior is that deep learning algorithms typically involve multiple sources of randomness whose individual contributions are not visible in the total variance. To enable fine-grained analysis, we describe an interpretable, symmetric decomposition of the variance into terms associated with the randomness from sampling, initialization, and the labels. Moreover, we compute the high-dimensional asymptotic behavior of this decomposition for random feature kernel regression, and analyze the strikingly rich phenomenology that arises. We find that the bias decreases monotonically with the network width, but the variance terms exhibit non-monotonic behavior and can diverge at the interpolation boundary, even in the absence of label noise. The divergence is caused by the interaction between sampling and initialization and can therefore be eliminated by marginalizing over samples (i.e. bagging) or over the initial parameters (i.e. ensemble learning).