mlsanityData to 14 Oct 2026

Reconciling modern machine learning practice and the bias-variance trade-off

arXiv 1812.11118 ↗First posted 28 Dec 2018stat.ML cs.LGcited by 927

Not on the map. Why?

Abstract (arXiv)

Breakthroughs in machine learning are rapidly changing science and society, yet our fundamental understanding of this technology has lagged far behind. Indeed, one of the central tenets of the field, the bias-variance trade-off, appears to be at odds with the observed behavior of methods used in the modern machine learning practice. The bias-variance trade-off implies that a model should balance under-fitting and over-fitting: rich enough to express underlying structure in data, simple enough to avoid fitting spurious patterns. However, in the modern practice, very rich models such as neural networks are trained to exactly fit (i.e., interpolate) the data. Classically, such models would be considered over-fit, and yet they often obtain high accuracy on test data. This apparent contradiction has raised questions about the mathematical foundations of machine learning and their relevance to practitioners. In this paper, we reconcile the classical understanding and the modern practice within a unified performance curve. This "double descent" curve subsumes the textbook U-shaped bias-variance trade-off curve by showing how increasing model capacity beyond the point of interpolation results in improved performance. We provide evidence for the existence and ubiquity of double descent for a wide spectrum of models and datasets, and we posit a mechanism for its emergence. This connection between the performance and the structure of machine learning models delineates the limits of classical analyses, and has implications for both the theory and practice of machine learning.

Key facts. Cited by 927 papers in the map (data through 14 Oct 2026). First posted on arXiv 28 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 161; 2022: 161; 2023: 159; 2024: 143; 2025: 155; 2026: 147.

927 papers.

Sort:

By area:

  1. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  2. 2022In-context Learning and Induction Headscited by 964
  3. 2023Large Language Modelscited by 780
  4. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  5. 2022Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's nextcited by 608
  6. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  7. 2021A Field Guide to Federated Optimizationcited by 315
  8. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  9. 2022U-NO: U-shaped Neural Operatorscited by 230
  10. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  11. 2023Better Diffusion Models Further Improve Adversarial Trainingcited by 208
  12. 2021Trustworthy AI: From Principles to Practicescited by 207
38 more
  1. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  2. 2021Neural Symbolic Regression that Scalescited by 192
  3. 2021Training Graph Neural Networks with 1000 Layerscited by 169
  4. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  5. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  6. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  7. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
  8. 2021A Universal Law of Robustness via Isoperimetrycited by 151
  9. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  10. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 148
  11. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 132
  12. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  13. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
  14. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 119
  15. 2021Assessing Generalization of SGD via Disagreementcited by 117
  16. 2022A Solvable Model of Neural Scaling Lawscited by 116
  17. 2021Understanding and Improving Early Stopping for Learning with Noisy Labelscited by 113
  18. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  19. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  20. 2022Robust Training under Label Noise by Over-parameterizationcited by 93
  21. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 91
  22. 2025Model Organisms for Emergent Misalignmentcited by 91
  23. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  24. 2021Learning Curve Theorycited by 85
  25. 2021ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reductioncited by 84
  26. 2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 76
  27. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  28. 2021The Shape of Learning Curves: a Reviewcited by 70
  29. 2024Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilitiescited by 69
  30. 2022Deep Ensembles Work, But Are They Necessary?cited by 63
  31. 2021Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracycited by 62
  32. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  33. 2022Reliable extrapolation of deep neural operators informed by physics or sparse observationscited by 62
  34. 2023Prediction-Oriented Bayesian Active Learningcited by 62
  35. 2022GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasetscited by 61
  36. 2025How much do language models memorize?cited by 61
  37. 2021On Memorization in Probabilistic Deep Generative Modelscited by 60
  38. 2021A Theory of PAC Learnability of Partial Concept Classescited by 60

50 most cited of 927 All 927 (JSON)