# Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime

arXiv: https://arxiv.org/abs/2003.01054 · HTML: https://mlsanity.com/paper/2003.01054/

## Key facts

- Cited by 95 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Mar 2020.

## Built on by (most cited first, 30 of 95)

- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks](https://mlsanity.com/paper/2006.13198/) (2020)
- [A Dynamical Model of Neural Scaling Laws](https://mlsanity.com/paper/2402.01092/) (2024)
- [Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Training](https://mlsanity.com/paper/2505.17638/) (2025)
- [A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning](https://mlsanity.com/paper/2109.02355/) (2021)
- [Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective](https://mlsanity.com/paper/2203.08124/) (2022)
- [Scaling and renormalization in high-dimensional regression](https://mlsanity.com/paper/2405.00592/) (2024)
- [Deterministic equivalent and error universality of deep random features learning](https://mlsanity.com/paper/2302.00401/) (2023)
- [Universality laws for Gaussian mixtures in generalized linear models](https://mlsanity.com/paper/2302.08933/) (2023)
- [Sparse Double Descent: Where Network Pruning Aggravates Overfitting](https://mlsanity.com/paper/2206.08684/) (2022)
- [Probing transfer learning with a model of synthetic correlated datasets](https://mlsanity.com/paper/2106.05418/) (2021)
- [The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimes](https://mlsanity.com/paper/2212.12147/) (2022)
- [Multi-scale Feature Learning Dynamics: Insights for Double Descent](https://mlsanity.com/paper/2112.03215/) (2021)
- [Contrasting random and learned features in deep Bayesian linear regression](https://mlsanity.com/paper/2203.00573/) (2022)
- [Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression](https://mlsanity.com/paper/2205.14846/) (2022)
- [Out-of-Distribution Generalization in Kernel Regression](https://mlsanity.com/paper/2106.02261/) (2021)
- [Learning through atypical "phase transitions" in overparameterized neural networks](https://mlsanity.com/paper/2110.00683/) (2021)
- [Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics for Convex Losses in High-Dimension](https://mlsanity.com/paper/2201.13383/) (2022)
- [Sharp Asymptotics of Kernel Ridge Regression Beyond the Linear Regime](https://mlsanity.com/paper/2205.06798/) (2022)
- [Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer](https://mlsanity.com/paper/2502.02531/) (2025)
- [Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature model](https://mlsanity.com/paper/2110.11805/) (2021)
- [On double-descent in uncertainty quantification in overparametrized models](https://mlsanity.com/paper/2210.12760/) (2022)
- [Pathologies of Predictive Diversity in Deep Ensembles](https://mlsanity.com/paper/2302.00704/) (2023)
- [AEDFL: Efficient Asynchronous Decentralized Federated Learning with Heterogeneous Devices](https://mlsanity.com/paper/2312.10935/) (2023)
- [Harmless interpolation in regression and classification with structured features](https://mlsanity.com/paper/2111.05198/) (2021)
- [Exact Gap between Generalization Error and Uniform Convergence in Random Feature Models](https://mlsanity.com/paper/2103.04554/) (2021)
- [On the Generalization Power of Overfitted Two-Layer Neural Tangent Kernel Models](https://mlsanity.com/paper/2103.05243/) (2021)
- [Learning curves for deep structured Gaussian feature models](https://mlsanity.com/paper/2303.00564/) (2023)
- [Fundamental limits of overparametrized shallow neural networks for supervised learning](https://mlsanity.com/paper/2307.05635/) (2023)
- [Understanding the bias-variance tradeoff of Bregman divergences](https://mlsanity.com/paper/2202.04167/) (2022)

## Abstract (arXiv)

Deep neural networks can achieve remarkable generalization performances while interpolating the training data perfectly. Rather than the U-curve emblematic of the bias-variance trade-off, their test error often follows a "double descent" - a mark of the beneficial role of overparametrization. In this work, we develop a quantitative theory for this phenomenon in the so-called lazy learning regime of neural networks, by considering the problem of learning a high-dimensional function with random features regression. We obtain a precise asymptotic expression for the bias-variance decomposition of the test error, and show that the bias displays a phase transition at the interpolation threshold, beyond which it remains constant. We disentangle the variances stemming from the sampling of the dataset, from the additive noise corrupting the labels, and from the initialization of the weights. Following up on Geiger et al. 2019, we first show that the latter two contributions are the crux of the double descent: they lead to the overfitting peak at the interpolation threshold and to the decay of the test error upon overparametrization. We then quantify how they are suppressed by ensemble averaging the outputs of K independently initialized estimators. When K is sent to infinity, the test error remains constant beyond the interpolation threshold. We further compare the effects of overparametrizing, ensembling and regularizing. Finally, we present numerical experiments on classic deep learning setups to show that our results hold qualitatively in realistic lazy learning scenarios.