mlsanityData to 7 Oct 2026

A Modern Take on the Bias-Variance Tradeoff in Neural Networks

arXiv 1810.08591 ↗First posted 19 Oct 2018cs.LG stat.MLcited by 64

Not on the map. Why?

Abstract (arXiv)

The bias-variance tradeoff tells us that as model complexity increases, bias falls and variances increases, leading to a U-shaped test error curve. However, recent empirical results with over-parameterized neural networks are marked by a striking absence of the classic U-shaped test error curve: test error keeps decreasing in wider networks. This suggests that there might not be a bias-variance tradeoff in neural networks with respect to network width, unlike was originally claimed by, e.g., Geman et al. (1992). Motivated by the shaky evidence used to support this claim in neural networks, we measure bias and variance in the modern setting. We find that both bias and variance can decrease as the number of parameters grows. To better understand this, we introduce a new decomposition of the variance to disentangle the effects of optimization and data sampling. We also provide theoretical analysis in a simplified setting that is consistent with our empirical findings.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 17; 2023: 13; 2024: 10; 2025: 8; 2026: 2.

64 papers.

Sort:

By area:

  1. 2021From Stars to Subgraphs: Uplifting Any GNN with Local Structure Awarenesscited by 154
  2. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  3. 2021Assessing Generalization of SGD via Disagreementcited by 116
  4. 2022Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspectivecited by 57
  5. 2022Packed-Ensembles for Efficient Uncertainty Estimationcited by 49
  6. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
  7. 2021MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworkscited by 32
  8. 2023A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learningcited by 32
  9. 2023Some Fundamental Aspects about Lipschitz Continuity of Neural Networkscited by 25
  10. 2021Multi-scale Feature Learning Dynamics: Insights for Double Descentcited by 24
  11. 2022Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics for Convex Losses in High-Dimensioncited by 19
  12. 2022Membership Inference Attacks and Generalization: A Causal Perspectivecited by 17
38 more
  1. 2023Pathologies of Predictive Diversity in Deep Ensemblescited by 16
  2. 2021Agree to Disagree: When Deep Learning Models With Identical Architectures Produce Distinct Explanationscited by 15
  3. 2023Rethinking Semi-Supervised Imbalanced Node Classification from Bias-Variance Decompositioncited by 13
  4. 2022Understanding the bias-variance tradeoff of Bregman divergencescited by 10
  5. 2021Redundant representations help generalization in wide neural networkscited by 9
  6. 2022Agent-Temporal Attention for Reward Redistribution in Episodic Multi-Agent Reinforcement Learningcited by 9
  7. 2022Regularization-wise double descent: Why it occurs and how to eliminate itcited by 9
  8. 2022Task Discovery: Finding the Tasks that Neural Networks Generalize oncited by 9
  9. 2024Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyondcited by 9
  10. 2022An Analysis on Ensemble Learning optimized Medical Image Classification with Deep Convolutional Neural Networkscited by 8
  11. 2022Deep Double Descent via Smooth Interpolationcited by 8
  12. 2022Ensembling over Classifiers: a Bias-Variance Perspectivecited by 6
  13. 2023Evaluating Self-Supervised Learning via Risk Decompositioncited by 6
  14. 2024Understanding Model Ensemble in Transferable Adversarial Attackcited by 6
  15. 2025Bias-variance decompositions: the exclusive privilege of Bregman divergencescited by 6
  16. 2023Training-Free Neural Active Learning with Initialization-Robustness Guaranteescited by 5
  17. 2023Physics-informed neural wavefields with Gabor basis functionscited by 5
  18. 2024Breaking Neural Network Scaling Laws with Modularitycited by 5
  19. 2024Classical Statistical (In-Sample) Intuitions Don't Generalize Well: A Note on Bias-Variance Tradeoffs, Overfitting and Moving from Fixed to Random Designscited by 5
  20. 2021On the privacy-utility trade-off in differentially private hierarchical text classificationcited by 4
  21. 2021Optimization Variance: Exploring Generalization Properties of DNNscited by 4
  22. 2021Topologically-Informed Atlas Learningcited by 4
  23. 2022VC Theoretical Explanation of Double Descentcited by 4
  24. 2021Mitigating deep double descent by concatenating inputscited by 3
  25. 2022The Dynamic of Consensus in Deep Networks and the Identification of Noisy Labelscited by 3
  26. 2023Can we learn better with hard samples?cited by 3
  27. 2023Does Double Descent Occur in Self-Supervised Learning?cited by 3
  28. 2023Reflection coupling for unadjusted generalized Hamiltonian Monte Carlo in the nonconvex stochastic gradient casecited by 3
  29. 2024Assessment of Uncertainty Quantification in Universal Differential Equationscited by 3
  30. 2024Concept learning of parameterized quantum models from limited measurementscited by 3
  31. 2025Gabor-Enhanced Physics-Informed Neural Networks for Fast Simulations of Acoustic Wavefieldscited by 3
  32. 2025Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuningcited by 3
  33. 2022Bias-Variance Decompositions for Margin Lossescited by 2
  34. 2022Understanding the double descent curve in Machine Learningcited by 2
  35. 2024Over-parameterization and Adversarial Robustness in Neural Networks: An Overview and Empirical Analysiscited by 2
  36. 2024Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networkscited by 2
  37. 2021More layers! End-to-end regression and uncertainty on tabular data with deep learningcited by 1
  38. 2022The BUTTER Zone: An Empirical Study of Training Dynamics in Fully Connected Neural Networkscited by 1

50 most cited of 64 All 64 (JSON)