mlsanityData to 7 Oct 2026

Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionality

arXiv 2409.03891 ↗First posted 5 Sep 2024cs.LG stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

We consider the overfitting behavior of minimum norm interpolating solutions of Gaussian kernel ridge regression (i.e. kernel ridgeless regression), when the bandwidth or input dimension varies with the sample size. For fixed dimensions, we show that even with varying or tuned bandwidth, the ridgeless solution is never consistent and, at least with large enough noise, always worse than the null predictor. For increasing dimension, we give a generic characterization of the overfitting behavior for any scaling of the dimension with sample size. We use this to provide the first example of benign overfitting using the Gaussian kernel with sub-polynomial scaling dimension. All our results are under the Gaussian universality ansatz and the (non-rigorous) risk predictions in terms of the kernel eigenstructure.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 55 of its 69 references that are in the map. First posted on arXiv 5 Sep 2024.

Builds on

55 of 69 references.

References · 55

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2019Benign Overfitting in Linear Regressioncited by 544
  3. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  4. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  5. 2019Two models of double descent for weak featurescited by 194
  6. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
  7. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  8. 2019Linearized two-layers neural networks in high dimensioncited by 155
  9. 2020Benign overfitting in ridge regressioncited by 152
  10. 2019Harmless interpolation of noisy data in regressioncited by 121
  11. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  12. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
38 more
  1. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  2. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  3. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
  4. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  5. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  6. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  7. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
  8. 2022Dimension free ridge regressioncited by 50
  9. 2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 48
  10. 2023High-dimensional analysis of double descent for linear regression with random projectionscited by 47
  11. 2018Consistency of Interpolation with Laplace Kernels is a High-Dimensional Phenomenoncited by 44
  12. 2019In Defense of Uniform Convergence: Generalization via derandomization with an application to interpolating predictorscited by 44
  13. 2022Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regressioncited by 42
  14. 2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 41
  15. 2022The Implicit Bias of Benign Overfittingcited by 38
  16. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  17. 2020Theoretical Insights Into Multiclass Classification: A High-dimensional Asymptotic Viewcited by 32
  18. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 32
  19. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  20. 2019The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regimecited by 27
  21. 2020Binary Classification of Gaussian Mixtures: Abundance of Support Vectors, Benign Overfitting and Regularizationcited by 27
  22. 2020Failures of model-dependent generalization bounds for least-norm interpolationcited by 26
  23. 2023Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimensioncited by 26
  24. 2023Generalization in Kernel Regression Under Realistic Assumptionscited by 26
  25. 2021The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networkscited by 25
  26. 2021Interpolating Classifiers Make Few Mistakescited by 22
  27. 2022Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regressioncited by 22
  28. 2024Asymptotics of Random Feature Regression Beyond the Linear Scaling Regimecited by 22
  29. 2024A non-asymptotic theory of Kernel Ridge Regression: deterministic equivalents, test error, and GCV estimatorcited by 22
  30. 2022Fast Rates for Noisy Interpolation Require Rethinking the Effects of Inductive Biascited by 21
  31. 2021The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networkscited by 18
  32. 2021Tight bounds for minimum l1-norm interpolation of noisy datacited by 16
  33. 2023From Tempered to Benign Overfitting in ReLU Neural Networkscited by 16
  34. 2022On the Inconsistency of Kernel Ridgeless Regression in Fixed Dimensionscited by 13
  35. 2024Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrumcited by 13
  36. 2022A Non-Asymptotic Moreau Envelope Theory for High-Dimensional Generalized Linear Modelscited by 11
  37. 2023Noisy Interpolation Learning with Shallow Univariate ReLU Networkscited by 11
  38. 2023Benign Overfitting in Two-Layer ReLU Convolutional Neural Networks for XOR Datacited by 10

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 7.

7 papers.

  1. 2025Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approachescited by 4
  2. 2025When Bias Pretends to Be Truth: How Spurious Correlations Undermine Hallucination Detection in LLMscited by 3
  3. 2025Beyond Benign Overfitting in Nadaraya-Watson Interpolatorscited by 2
  4. 2025Transfer Learning for Benign Overfitting in High-Dimensional Linear Regressioncited by 2
  5. 2025Benign Overfitting with Quantum Kernelscited by 1
  6. 2025Rasterized Steered Mixture of Experts for Efficient 2D Image Regressioncited by 0
  7. 2025Calibrated Principal Component Regressioncited by 0