mlsanityData to 14 Oct 2026

More Data Can Hurt for Linear Regression: Sample-wise Double Descent

arXiv 1912.07242 ↗First posted 16 Dec 2019stat.ML cs.LG cs.NEcited by 37

Not on the map. Why?

Abstract (arXiv)

In this expository note we describe a surprising phenomenon in overparameterized linear regression, where the dimension exceeds the number of samples: there is a regime where the test risk of the estimator found by gradient descent increases with additional samples. In other words, more data actually hurts the estimator. This behavior is implicit in a recent line of theoretical works analyzing "double-descent" phenomenon in linear models. In this note, we isolate and understand this behavior in an extremely simple setting: linear regression with isotropic Gaussian covariates. In particular, this occurs due to an unconventional type of bias-variance tradeoff in the overparameterized regime: the bias decreases with more samples, but variance increases.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 9; 2022: 13; 2023: 4; 2024: 5; 2025: 3; 2026: 2.

37 papers.

Sort:

By area:

  1. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  2. 2021Explaining Neural Scaling Lawscited by 363
  3. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  4. 2021The Shape of Learning Curves: a Reviewcited by 70
  5. 2024Scaling and renormalization in high-dimensional regressioncited by 50
  6. 2022Contrasting random and learned features in deep Bayesian linear regressioncited by 24
  7. 2024Can In-context Learning Really Generalize to Out-of-distribution Tasks?cited by 23
  8. 2021Out-of-Distribution Generalization in Kernel Regressioncited by 21
  9. 2021When and how epochwise double descent happenscited by 21
  10. 2022Phenomenology of Double Descent in Finite-Width Neural Networkscited by 12
  11. 2021Meta-learning with negative learning ratescited by 11
  12. 2022Differentially Private Regression with Unbounded Covariatescited by 10
25 more
  1. 2022Mitigating multiple descents: A model-agnostic framework for risk monotonizationcited by 10
  2. 2026Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentioncited by 10
  3. 2022Regularization-wise double descent: Why it occurs and how to eliminate itcited by 9
  4. 2022Generalization for multiclass classification with overparameterized linear modelscited by 8
  5. 2022Bias-variance decomposition of overparameterized regression with random linear featurescited by 7
  6. 2024How more data can hurt: Instability and regularization in next-generation reservoir computingcited by 5
  7. 2021The Geometry of Over-parameterized Regression and Adversarial Perturbationscited by 4
  8. 2025Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scalingcited by 4
  9. 2021Mitigating deep double descent by concatenating inputscited by 3
  10. 2022Generalization and Overfitting in Matrix Product State Machine Learning Architecturescited by 3
  11. 2023Does Double Descent Occur in Self-Supervised Learning?cited by 3
  12. 2025Transformers Don't In-Context Learn Least Squares Regressioncited by 3
  13. 2022A Survey of Learning Curves with Bad Behavior: or How More Data Need Not Lead to Better Performancecited by 2
  14. 2023Collaborative Development of NLP modelscited by 2
  15. 2024A solvable generative model with a linear, one-step denoisercited by 2
  16. 2021Classification and Adversarial examples in an Overparameterized Linear Model: A Signal Processing Perspectivecited by 1
  17. 2022Information FOMO: The unhealthy fear of missing out on information. A method for removing misleading data for healthier modelscited by 1
  18. 2023Learning Curves for Noisy Heterogeneous Feature-Subsampled Ridge Ensemblescited by 1
  19. 2024No Free Lunch From Random Feature Ensembles: Scaling Laws and Near-Optimality Conditionscited by 1
  20. 2021Local Linearity and Double Descent in Catastrophic Overfittingcited by 0
  21. 2022Counterbalancing Teacher: Regularizing Batch Normalized Models for Robustnesscited by 0
  22. 2022High Dimensional Binary Classification under Label Shift: Phase Transition and Regularizationcited by 0
  23. 2023Gibbs-Based Information Criteria and the Over-Parameterized Regimecited by 0
  24. 2025The Double Descent Behavior in Two Layer Neural Network for Binary Classificationcited by 0
  25. 2026Mitigating Over-Optimization in PRM-Guided Search in Mathematical Reasoning by Optimizing the Guidecited by 0