Data to 7 Oct 2026

Surprises in High-Dimensional Ridgeless Least Squares Interpolation

arXiv 1903.08560 ↗First posted 19 Mar 2019math.ST cs.LG stat.MLcited by 508

Not on the map. Why?

Abstract (arXiv)

Interpolators -- estimators that achieve zero training error -- have attracted growing attention in machine learning, mainly because state-of-the art neural networks appear to be models of this type. In this paper, we study minimum ℓ2 norm ("ridgeless") interpolation in high-dimensional least squares regression. We consider two different models for the feature distribution: a linear model, where the feature vectors xi ∈ mathbb Rp are obtained by applying a linear transform to a vector of i.i.d. entries, xi = Sigma1/2 zi (with zi ∈ mathbb Rp); and a nonlinear model, where the feature vectors are obtained by passing the input through a random one-layer neural network, xi = varphi(W zi) (with zi ∈ mathbb Rd, W ∈ mathbb Rp × d a matrix of i.i.d. entries, and varphi an activation function acting componentwise on W zi). We recover -- in a precise quantitative way -- several phenomena that have been observed in large-scale neural networks and kernel machines, including the "double descent" behavior of the prediction risk, and the potential benefits of overparametrization.

Key facts. Cited by 508 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 89; 2022: 100; 2023: 97; 2024: 77; 2025: 80; 2026: 64.

508 papers.

Sort:

By area:

2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
2021Explaining Neural Scaling Lawscited by 353
2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
2021Dash: Semi-Supervised Learning with Dynamic Thresholdingcited by 147
2022If Influence Functions are the Answer, Then What is the Question?cited by 142
2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
2024A Dynamical Model of Neural Scaling Lawscited by 105
2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
2021Provably Strict Generalisation Benefit for Equivariant Modelscited by 90
2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90

38 more

2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
2024Model Collapse Demystified: The Case of Regressioncited by 80
2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
2022Universality of empirical risk minimizationcited by 73
2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
2023Theory on Forgetting and Generalization of Continual Learningcited by 69
2021The Shape of Learning Curves: a Reviewcited by 67
2021Cross-validation: what does it estimate and how well does it do it?cited by 65
2022Deep Ensembles Work, But Are They Necessary?cited by 63
2024Strong Model Collapsecited by 62
2021When and How Mixup Improves Calibrationcited by 59
2022Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspectivecited by 57
2025Deep Learning is Not So Mysterious or Differentcited by 55
2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
2021A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learningcited by 54
2024Asymptotic theory of in-context learning by linear attentioncited by 54
2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 52
2022Predicting Out-of-Distribution Error with the Projection Normcited by 50
2022Dimension free ridge regressioncited by 50
2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 48
2023High-dimensional analysis of double descent for linear regression with random projectionscited by 47
2024Scaling and renormalization in high-dimensional regressioncited by 45
2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
2022Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regressioncited by 42
2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 41
2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
2021Strength of Minibatch Noise in SGDcited by 39
2022The Implicit Bias of Benign Overfittingcited by 38
2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
2023Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzlecited by 36
2023Deep networks for system identification: a Surveycited by 35
2023Deterministic equivalent and error universality of deep random features learningcited by 35
2023Benign Overfitting for Two-layer ReLU Convolutional Neural Networkscited by 34
2021SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticalitycited by 33
2021KNAS: Green Neural Architecture Searchcited by 33
2021How rotational invariance of common kernels prevents generalization in high dimensionscited by 32
2021The Benefits of Implicit Regularization from SGD in Least Squares Problemscited by 32

50 most cited of 508 All 508 (JSON)