mlsanityData to 14 Oct 2026

Deep Double Descent: Where Bigger Models and More Data Hurt

arXiv 1912.02292 ↗First posted 4 Dec 2019cs.LG cs.CV cs.NEcited by 672

Not on the map. Why?

Abstract (arXiv)

We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better. Moreover, we show that double descent occurs not just as a function of model size, but also as a function of the number of training epochs. We unify the above phenomena by defining a new complexity measure we call the effective model complexity and conjecture a generalized double descent with respect to this measure. Furthermore, our notion of model complexity allows us to identify certain regimes where increasing (even quadrupling) the number of train samples actually hurts test performance.

Key facts. Cited by 672 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 120; 2022: 134; 2023: 119; 2024: 122; 2025: 93; 2026: 83.

672 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2022In-context Learning and Induction Headscited by 964
  3. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  4. 2023Scaling Data-Constrained Language Modelscited by 403
  5. 2021Explaining Neural Scaling Lawscited by 363
  6. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
  7. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  8. 2021Scaling Laws for Transfercited by 272
  9. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
  10. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  11. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
  12. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
38 more
  1. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 195
  2. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  3. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
  4. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 152
  5. 2021A Universal Law of Robustness via Isoperimetrycited by 151
  6. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 148
  7. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  8. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 146
  9. 2023Taken out of context: On measuring situational awareness in LLMscited by 143
  10. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  11. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 132
  12. 2023End-to-End Speech Recognition: A Surveycited by 129
  13. 2023On Memorization in Diffusion Modelscited by 125
  14. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 118
  15. 2021Assessing Generalization of SGD via Disagreementcited by 117
  16. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  17. 2022A Solvable Model of Neural Scaling Lawscited by 116
  18. 2021Understanding and Improving Early Stopping for Learning with Noisy Labelscited by 113
  19. 2022Broken Neural Scaling Lawscited by 109
  20. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  21. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  22. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  23. 2022Training Trajectories of Language Models Across Scalescited by 81
  24. 2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 76
  25. 2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 76
  26. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 76
  27. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  28. 2021On the geometry of generalization and memorization in deep neural networkscited by 73
  29. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 72
  30. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  31. 2021The Shape of Learning Curves: a Reviewcited by 70
  32. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  33. 2021Model Complexity of Deep Learning: A Surveycited by 65
  34. 2022Deep Ensembles Work, But Are They Necessary?cited by 63
  35. 2021Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracycited by 62
  36. 2022Reliable extrapolation of deep neural operators informed by physics or sparse observationscited by 62
  37. 2025How much do language models memorize?cited by 61
  38. 2021On Memorization in Probabilistic Deep Generative Modelscited by 60

50 most cited of 672 All 672 (JSON)