mlsanityData to 7 Oct 2026

Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrum

arXiv 2402.01297 ↗First posted 2 Feb 2024cs.LG stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

We derive new bounds for the condition number of kernel matrices, which we then use to enhance existing non-asymptotic test error bounds for kernel ridgeless regression (KRR) in the over-parameterized regime for a fixed input dimension. For kernels with polynomial spectral decay, we recover the bound from previous work; for exponential decay, our bound is non-trivial and novel. Our contribution is two-fold: (i) we rigorously prove the phenomena of tempered overfitting and catastrophic overfitting under the sub-Gaussian design assumption, closing an existing gap in the literature; (ii) we identify that the independence of the features plays an important role in guaranteeing tempered overfitting, raising concerns about approximating KRR generalization using the Gaussian design assumption in previous literature.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 23 of its 35 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

23 of 35 references.

References · 23

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  4. 2019Benign Overfitting in Linear Regressioncited by 544
  5. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  6. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  7. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  8. 2020Benign overfitting in ridge regressioncited by 152
  9. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  10. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  11. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  12. 2020Implicit Regularization of Random Feature Modelscited by 54
11 more
  1. 2023High-dimensional analysis of double descent for linear regression with random projectionscited by 47
  2. 2022Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regressioncited by 42
  3. 2020Kernel regression in high dimensions: Refined analysis beyond double descentcited by 39
  4. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  5. 2023On the Optimality of Misspecified Spectral Algorithmscited by 28
  6. 2023Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimensioncited by 26
  7. 2023Generalization in Kernel Regression Under Realistic Assumptionscited by 26
  8. 2021The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networkscited by 25
  9. 2023On the Asymptotic Learning Curves of Kernel Ridge Regression under Power-law Decaycited by 19
  10. 2023Kernel interpolation generalizes poorlycited by 16
  11. 2023A Theoretical Analysis of the Test Error of Finite-Rank Kernel Ridge Regressioncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 6; 2026: 1.

13 papers.

Sort:

  1. 2024The phase diagram of kernel interpolation in large dimensionscited by 9
  2. 2024Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionalitycited by 7
  3. 2024A Comprehensive Analysis on the Learning Curve in Kernel Ridge Regressioncited by 6
  4. 2025When Bias Pretends to Be Truth: How Spurious Correlations Undermine Hallucination Detection in LLMscited by 3
  5. 2025Beyond Benign Overfitting in Nadaraya-Watson Interpolatorscited by 2
  6. 2025Benign Overfitting with Quantum Kernelscited by 1
  7. 2025Online Federation For Mixtures of Proprietary Agents with Black-Box Encoderscited by 1
  8. 2025Beyond Universal Approximation Theorems: Algorithmic Uniform Approximation by Neural Networks Trained with Noisy Datacited by 1
  9. 2024Tighter Learning Guarantees on Digital Computers via Concentration of Measure on Finite Spacescited by 0
  10. 2024Benign overfitting in Fixed Dimension via Physics-Informed Learning with Smooth Inductive Biascited by 0
  11. 2024Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theoriescited by 0
  12. 2025Spectral Analysis of Molecular Features: When Richer Features Do Not Guarantee Better Generalizationcited by 0
1 more
  1. 2026Harmful Overfitting in Sobolev Spacescited by 0