mlsanityData to 14 Oct 2026

An Agnostic View on the Cost of Overfitting in (Kernel) Ridge Regression

arXiv 2306.13185 ↗First posted 22 Jun 2023stat.ML cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

We study the cost of overfitting in noisy kernel ridge regression (KRR), which we define as the ratio between the test error of the interpolating ridgeless model and the test error of the optimally-tuned model. We take an "agnostic" view in the following sense: we consider the cost as a function of sample size for any target function, even if the sample size is not large enough for consistency or the target is outside the RKHS. We analyze the cost of overfitting under a Gaussian universality ansatz using recently derived (non-rigorous) risk estimates in terms of the task eigenstructure. Our analysis provides a more refined characterization of benign, tempered and catastrophic overfitting (cf. Mallinar et al. 2022).

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 22 Jun 2023.

Builds on

28 of 40 references.

References · 28

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  2. 2019Benign Overfitting in Linear Regressioncited by 549
  3. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  4. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  5. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  6. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  7. 2019Linearized two-layers neural networks in high dimensioncited by 155
  8. 2020Benign overfitting in ridge regressioncited by 153
  9. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  10. 2019Harmless interpolation of noisy data in regressioncited by 121
  11. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  12. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
16 more
  1. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 98
  2. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 91
  3. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  4. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 62
  5. 2020Kernel Alignment Risk Estimator: Risk Prediction from Training Datacited by 54
  6. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
  7. 2022Dimension free ridge regressioncited by 51
  8. 2023High-dimensional analysis of double descent for linear regression with random projectionscited by 47
  9. 2022Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regressioncited by 45
  10. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  11. 2021The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networkscited by 26
  12. 2020On Uniform Convergence and Low-Norm Interpolation Learningcited by 24
  13. 2022Fast Rates for Noisy Interpolation Require Rethinking the Effects of Inductive Biascited by 22
  14. 2021Optimistic Rates: A Unifying Theory for Interpolation Learning and Regularization in Linear Regressioncited by 17
  15. 2021Tight bounds for minimum l1-norm interpolation of noisy datacited by 16
  16. 2022A Non-Asymptotic Moreau Envelope Theory for High-Dimensional Generalized Linear Modelscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 0; 2026: 1.

5 papers.

  1. 2023More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatorycited by 18
  2. 2024Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionalitycited by 7
  3. 2024Provable Tempered Overfitting of Minimal Nets and Typical Netscited by 5
  4. 2024Revisiting Optimism and Model Complexity in the Wake of Overparameterized Machine Learningcited by 2
  5. 2026A mathematical theory of balancing relational generalization and memorizationcited by 0