# More Data Can Hurt for Linear Regression: Sample-wise Double Descent

arXiv: https://arxiv.org/abs/1912.07242 · HTML: https://mlsanity.com/paper/1912.07242/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Dec 2019.

## Built on by (most cited first, 30 of 37)

- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks](https://mlsanity.com/paper/2006.13198/) (2020)
- [The Shape of Learning Curves: a Review](https://mlsanity.com/paper/2103.10948/) (2021)
- [Scaling and renormalization in high-dimensional regression](https://mlsanity.com/paper/2405.00592/) (2024)
- [Contrasting random and learned features in deep Bayesian linear regression](https://mlsanity.com/paper/2203.00573/) (2022)
- [Can In-context Learning Really Generalize to Out-of-distribution Tasks?](https://mlsanity.com/paper/2410.09695/) (2024)
- [Out-of-Distribution Generalization in Kernel Regression](https://mlsanity.com/paper/2106.02261/) (2021)
- [When and how epochwise double descent happens](https://mlsanity.com/paper/2108.12006/) (2021)
- [Phenomenology of Double Descent in Finite-Width Neural Networks](https://mlsanity.com/paper/2203.07337/) (2022)
- [Meta-learning with negative learning rates](https://mlsanity.com/paper/2102.00940/) (2021)
- [Differentially Private Regression with Unbounded Covariates](https://mlsanity.com/paper/2202.11199/) (2022)
- [Mitigating multiple descents: A model-agnostic framework for risk monotonization](https://mlsanity.com/paper/2205.12937/) (2022)
- [Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention](https://mlsanity.com/paper/2605.29548/) (2026)
- [Regularization-wise double descent: Why it occurs and how to eliminate it](https://mlsanity.com/paper/2206.01378/) (2022)
- [Generalization for multiclass classification with overparameterized linear models](https://mlsanity.com/paper/2206.01399/) (2022)
- [Bias-variance decomposition of overparameterized regression with random linear features](https://mlsanity.com/paper/2203.05443/) (2022)
- [How more data can hurt: Instability and regularization in next-generation reservoir computing](https://mlsanity.com/paper/2407.08641/) (2024)
- [The Geometry of Over-parameterized Regression and Adversarial Perturbations](https://mlsanity.com/paper/2103.14108/) (2021)
- [Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling](https://mlsanity.com/paper/2512.19905/) (2025)
- [Mitigating deep double descent by concatenating inputs](https://mlsanity.com/paper/2107.00797/) (2021)
- [Generalization and Overfitting in Matrix Product State Machine Learning Architectures](https://mlsanity.com/paper/2208.04372/) (2022)
- [Does Double Descent Occur in Self-Supervised Learning?](https://mlsanity.com/paper/2307.07872/) (2023)
- [Transformers Don't In-Context Learn Least Squares Regression](https://mlsanity.com/paper/2507.09440/) (2025)
- [A Survey of Learning Curves with Bad Behavior: or How More Data Need Not Lead to Better Performance](https://mlsanity.com/paper/2211.14061/) (2022)
- [Collaborative Development of NLP models](https://mlsanity.com/paper/2305.12219/) (2023)
- [A solvable generative model with a linear, one-step denoiser](https://mlsanity.com/paper/2411.17807/) (2024)
- [Classification and Adversarial examples in an Overparameterized Linear Model: A Signal Processing Perspective](https://mlsanity.com/paper/2109.13215/) (2021)
- [Information FOMO: The unhealthy fear of missing out on information. A method for removing misleading data for healthier models](https://mlsanity.com/paper/2208.13080/) (2022)
- [Learning Curves for Noisy Heterogeneous Feature-Subsampled Ridge Ensembles](https://mlsanity.com/paper/2307.03176/) (2023)

## Abstract (arXiv)

In this expository note we describe a surprising phenomenon in overparameterized linear regression, where the dimension exceeds the number of samples: there is a regime where the test risk of the estimator found by gradient descent increases with additional samples. In other words, more data actually hurts the estimator. This behavior is implicit in a recent line of theoretical works analyzing "double-descent" phenomenon in linear models. In this note, we isolate and understand this behavior in an extremely simple setting: linear regression with isotropic Gaussian covariates. In particular, this occurs due to an unconventional type of bias-variance tradeoff in the overparameterized regime: the bias decreases with more samples, but variance increases.