Data to 14 Oct 2026

Benign Overfitting in Linear Regression

arXiv 1906.11300 ↗First posted 26 Jun 2019stat.ML cs.LG math.STcited by 549

Not on the map. Why?

Abstract (arXiv)

The phenomenon of benign overfitting is one of the key mysteries uncovered by deep learning methodology: deep neural networks seem to predict well, even with a perfect fit to noisy training data. Motivated by this phenomenon, we consider when a perfect fit to training data in linear regression is compatible with accurate prediction. We give a characterization of linear regression problems for which the minimum norm interpolating prediction rule has near-optimal prediction accuracy. The characterization is in terms of two notions of the effective rank of the data covariance. It shows that overparameterization is essential for benign overfitting in this setting: the number of directions in parameter space that are unimportant for prediction must significantly exceed the sample size. By studying examples of data covariance properties that this characterization shows are required for benign overfitting, we find an important role for finite-dimensional data: the accuracy of the minimum norm interpolating prediction rule approaches the best possible accuracy for a much narrower range of properties of the data distribution when the data lies in an infinite dimensional space versus when the data lies in a finite dimensional space whose dimension grows faster than the sample size.

Key facts. Cited by 549 papers in the map (data through 14 Oct 2026). First posted on arXiv 26 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 89; 2022: 105; 2023: 104; 2024: 76; 2025: 83; 2026: 92.

549 papers.

Sort:

By area:

2023Large Language Modelscited by 780
2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
2021A Universal Law of Robustness via Isoperimetrycited by 151
2022If Influence Functions are the Answer, Then What is the Question?cited by 144
2022Training Data Influence Analysis and Estimation: A Surveycited by 135
2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 132
2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
2023On Memorization in Diffusion Modelscited by 125
2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110

38 more

2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 96
2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 91
2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 91
2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
2023Transformers as Support Vector Machinescited by 74
2023Theory on Forgetting and Generalization of Continual Learningcited by 72
2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
2023Can Neural Network Memorization Be Localized?cited by 69
2024When Do We Not Need Larger Vision Models?cited by 67
2021A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learningcited by 57
2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 57
2025Deep Learning is Not So Mysterious or Differentcited by 56
2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 54
2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
2022Dimension free ridge regressioncited by 51
2022Predicting Out-of-Distribution Error with the Projection Normcited by 50
2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 49
2023High-dimensional analysis of double descent for linear regression with random projectionscited by 47
2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 44
2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 42
2021Last Iterate Risk Bounds of SGD with Decaying Stepsize for Overparameterized Linear Regressioncited by 38
2022The Implicit Bias of Benign Overfittingcited by 38
2022On the Power of Foundation Modelscited by 37
2023Deep networks for system identification: a Surveycited by 37
2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
2022A Law of Data Separation in Deep Learningcited by 36
2024How Does Unlabeled Data Provably Help Out-of-Distribution Detection?cited by 36
2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 35
2025Tracing the Representation Geometry of Language Models from Pretraining to Post-trainingcited by 35
2021Early-stopped neural networks are consistentcited by 34
2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
2023Benign Overfitting for Two-layer ReLU Convolutional Neural Networkscited by 34
2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 33
2021How rotational invariance of common kernels prevents generalization in high dimensionscited by 32
2021The Benefits of Implicit Regularization from SGD in Least Squares Problemscited by 32
2023WL meet VCcited by 31

50 most cited of 549 All 549 (JSON)