mlsanityData to 14 Oct 2026

On the Inconsistency of Kernel Ridgeless Regression in Fixed Dimensions

arXiv 2205.13525 ↗First posted 26 May 2022cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

``Benign overfitting'', the ability of certain algorithms to interpolate noisy training data and yet perform well out-of-sample, has been a topic of considerable recent interest. We show, using a fixed design setup, that an important class of predictors, kernel machines with translation-invariant kernels, does not exhibit benign overfitting in fixed dimensions. In particular, the estimated predictor does not converge to the ground truth with increasing sample size, for any non-zero regression function and any (even adaptive) bandwidth selection. To prove these results, we give exact expressions for the generalization error, and its decomposition in terms of an approximation error and an estimation error that elicits a trade-off based on the selection of the kernel bandwidth. Our results apply to commonly used translation-invariant kernels such as Gaussian, Laplace, and Cauchy.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 15 of its 27 references that are in the map. First posted on arXiv 26 May 2022.

Builds on

15 of 27 references.

References · 15

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  3. 2019Benign Overfitting in Linear Regressioncited by 549
  4. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  5. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  6. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  7. 2019Two models of double descent for weak featurescited by 195
  8. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  9. 2018To understand deep learning we need to understand kernel learningcited by 187
  10. 2019Harmless interpolation of noisy data in regressioncited by 121
  11. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  12. 2018Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolatecited by 99
3 more
  1. 2018Does data interpolation contradict statistical optimality?cited by 95
  2. 2018Consistency of Interpolation with Laplace Kernels is a High-Dimensional Phenomenoncited by 44
  3. 2021How rotational invariance of common kernels prevents generalization in high dimensionscited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 4; 2026: 2.

13 papers.

Sort:

  1. 2023From Tempered to Benign Overfitting in ReLU Neural Networkscited by 16
  2. 2024Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionalitycited by 7
  3. 2024On the Consistency of Kernel Methods with Dependent Observationscited by 4
  4. 2024Learning Analysis of Kernel Ridgeless Regression with Asymmetric Kernel Learningcited by 2
  5. 2024On the Pinsker bound of inner product kernel regression in large dimensionscited by 2
  6. 2025Beyond Benign Overfitting in Nadaraya-Watson Interpolatorscited by 2
  7. 2025Benign Overfitting with Quantum Kernelscited by 2
  8. 2024Generalization error of spectral algorithmscited by 1
  9. 2025Feature maps for the Laplacian kernel and its generalizationscited by 1
  10. 2026Harmful Overfitting in Sobolev Spacescited by 1
  11. 2024Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theoriescited by 0
  12. 2025Sobolev norm inconsistency of kernel interpolationcited by 0
1 more
  1. 2026Large Dimensional Kernel Ridge Regression: Extending to Product Kernelscited by 0