mlsanityData to 7 Oct 2026

Harmless interpolation of noisy data in regression

arXiv 1903.09139 ↗First posted 21 Mar 2019cs.LG stat.MLcited by 121

Not on the map. Why?

Abstract (arXiv)

A continuing mystery in understanding the empirical success of deep neural networks is their ability to achieve zero training error and generalize well, even when the training data is noisy and there are more parameters than data points. We investigate this overparameterized regime in linear regression, where all solutions that minimize training error interpolate the data, including noise. We characterize the fundamental generalization (mean-squared) error of any interpolating solution in the presence of noise, and show that this error decays to zero with the number of features. Thus, overparameterization can be explicitly beneficial in ensuring harmless interpolation of noise. We discuss two root causes for poor generalization that are complementary in nature -- signal "bleeding" into a large number of alias features, and overfitting of noise by parsimonious feature selectors. For the sparse linear model with noise, we provide a hybrid interpolating scheme that mitigates both these issues and achieves order-optimal MSE over all possible interpolating solutions.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 33; 2022: 31; 2023: 21; 2024: 18; 2025: 9; 2026: 9.

121 papers.

Sort:

By area:

  1. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  2. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  3. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  4. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  5. 2023Theory on Forgetting and Generalization of Continual Learningcited by 69
  6. 2022Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspectivecited by 57
  7. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
  8. 2021A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learningcited by 54
  9. 2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 48
  10. 2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 41
  11. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  12. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
38 more
  1. 2024Implicit Bias and Fast Convergence Rates for Self-attentioncited by 31
  2. 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
  3. 2021Towards an Understanding of Benign Overfitting in Neural Networkscited by 29
  4. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  5. 2022On the Impossible Safety of Large AI Modelscited by 27
  6. 2021Foolish Crowds Support Benign Overfittingcited by 24
  7. 2022The good, the bad and the ugly sides of data augmentation: An implicit spectral regularization perspectivecited by 24
  8. 2022Towards Sample-efficient Overparameterized Meta-learningcited by 21
  9. 2022Fast Rates for Noisy Interpolation Require Rethinking the Effects of Inductive Biascited by 21
  10. 2022Overparameterized Linear Regression under Adversarial Attackscited by 21
  11. 2021The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networkscited by 18
  12. 2021Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature modelcited by 17
  13. 2021Optimistic Rates: A Unifying Theory for Interpolation Learning and Regularization in Linear Regressioncited by 17
  14. 2023More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatorycited by 17
  15. 2021Tight bounds for minimum l1-norm interpolation of noisy datacited by 16
  16. 2024Provable Weak-to-Strong Generalization via Benign Overfittingcited by 16
  17. 2024The Pitfalls of Memorization: When Memorization Hurts Generalizationcited by 16
  18. 2021Interpolation can hurt robust generalization even when there is no noisecited by 15
  19. 2021Harmless interpolation in regression and classification with structured featurescited by 14
  20. 2022Memorize to Generalize: on the Necessity of Interpolation in High Dimensional Linear Regressioncited by 14
  21. 2021On the Generalization Power of Overfitted Two-Layer Neural Tangent Kernel Modelscited by 13
  22. 2022On the Inconsistency of Kernel Ridgeless Regression in Fixed Dimensionscited by 13
  23. 2022Phenomenology of Double Descent in Finite-Width Neural Networkscited by 11
  24. 2022Understanding Benign Overfitting in Gradient-Based Meta Learningcited by 11
  25. 2022A Non-Asymptotic Moreau Envelope Theory for High-Dimensional Generalized Linear Modelscited by 11
  26. 2023Subsample Ridge Ensembles: Equivalences and Generalized Cross-Validationcited by 11
  27. 2023Noisy Interpolation Learning with Shallow Univariate ReLU Networkscited by 11
  28. 2022Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processescited by 10
  29. 2023Optimal Rate of Kernel Regression in Large Dimensionscited by 10
  30. 2023Strong inductive biases provably prevent harmless interpolationcited by 9
  31. 2024On Goodhart's law, with an application to value alignmentcited by 9
  32. 2025Double descent in quantum kernel methodscited by 9
  33. 2021Support vector machines and linear regression coincide with very high-dimensional featurescited by 8
  34. 2022Generalization for multiclass classification with overparameterized linear modelscited by 8
  35. 2022Deep Linear Networks can Benignly Overfit when Shallow Ones Docited by 8
  36. 2022Deep Double Descent via Smooth Interpolationcited by 8
  37. 2022Bias-variance decomposition of overparameterized regression with random linear featurescited by 7
  38. 2022A Universal Trade-off Between the Model Size, Test Loss, and Training Loss of Linear Predictorscited by 7

50 most cited of 121 All 121 (JSON)