mlsanityData to 7 Oct 2026

Just Interpolate: Kernel "Ridgeless" Regression Can Generalize

arXiv 1808.00387 ↗First posted 1 Aug 2018math.ST cs.LG stat.MLcited by 190

Not on the map. Why?

Abstract (arXiv)

In the absence of explicit regularization, Kernel "Ridgeless" Regression with nonlinear kernels has the potential to fit the training data perfectly. It has been observed empirically, however, that such interpolated solutions can still generalize well on test data. We isolate a phenomenon of implicit regularization for minimum-norm interpolated solutions which is due to a combination of high dimensionality of the input data, curvature of the kernel function, and favorable geometric properties of the data such as an eigenvalue decay of the empirical covariance and kernel matrices. In addition to deriving a data-dependent upper bound on the out-of-sample error, we present experimental evidence suggesting that the phenomenon occurs in the MNIST dataset.

Key facts. Cited by 190 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 43; 2022: 36; 2023: 50; 2024: 29; 2025: 17; 2026: 14.

190 papers.

Sort:

By area:

  1. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  2. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  3. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  4. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  5. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  6. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  7. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  8. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  9. 2021Learning with invariances in random features and kernel modelscited by 89
  10. 2024Model Collapse Demystified: The Case of Regressioncited by 80
  11. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  12. 2023Transformers as Support Vector Machinescited by 67
38 more
  1. 2021A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learningcited by 54
  2. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
  3. 2022The Implicit Bias of Benign Overfittingcited by 38
  4. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  5. 2023Deep networks for system identification: a Surveycited by 35
  6. 2023Benign Overfitting for Two-layer ReLU Convolutional Neural Networkscited by 34
  7. 2021How rotational invariance of common kernels prevents generalization in high dimensionscited by 32
  8. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 32
  9. 2023On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domainscited by 30
  10. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  11. 2021Towards an Understanding of Benign Overfitting in Neural Networkscited by 29
  12. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  13. 2023Generalization Ability of Wide Neural Networks on ℝcited by 28
  14. 2023Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimensioncited by 26
  15. 2023Generalization in Kernel Regression Under Realistic Assumptionscited by 26
  16. 2021Multi-scale Feature Learning Dynamics: Insights for Double Descentcited by 24
  17. 2023Nonparametric regression using over-parameterized shallow ReLU neural networkscited by 24
  18. 2021Interpolating Classifiers Make Few Mistakescited by 22
  19. 2021Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networkscited by 21
  20. 2021The Three Stages of Learning Dynamics in High-Dimensional Kernel Methodscited by 21
  21. 2021Learning with convolution and pooling operations in kernel methodscited by 21
  22. 2021Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networkscited by 20
  23. 2023The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifoldcited by 20
  24. 2023Six Lectures on Linearized Neural Networkscited by 19
  25. 2023On the Asymptotic Learning Curves of Kernel Ridge Regression under Power-law Decaycited by 19
  26. 2021The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networkscited by 18
  27. 2022Cutting Some Slack for SGD with Adaptive Polyak Stepsizescited by 18
  28. 2022Sharp Asymptotics of Kernel Ridge Regression Beyond the Linear Regimecited by 18
  29. 2023More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatorycited by 17
  30. 2023Kernel interpolation generalizes poorlycited by 16
  31. 2023From Tempered to Benign Overfitting in ReLU Neural Networkscited by 16
  32. 2021Interpolation can hurt robust generalization even when there is no noisecited by 15
  33. 2021Harmless interpolation in regression and classification with structured featurescited by 14
  34. 2022Memorize to Generalize: on the Necessity of Interpolation in High Dimensional Linear Regressioncited by 14
  35. 2023A Spectral Theory of Neural Prediction and Alignmentcited by 14
  36. 2023Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Modelscited by 14
  37. 2021SVRG Meets AdaGrad: Painless Variance Reductioncited by 13
  38. 2021Exact Gap between Generalization Error and Uniform Convergence in Random Feature Modelscited by 13

50 most cited of 190 All 190 (JSON)