mlsanityData to 7 Oct 2026

On the Asymptotic Learning Curves of Kernel Ridge Regression under Power-law Decay

arXiv 2309.13337 ↗First posted 23 Sep 2023cs.LG math.ST stat.THcited by 19

Not on the map. Why?

Abstract (arXiv)

The widely observed 'benign overfitting phenomenon' in the neural network literature raises the challenge to the 'bias-variance trade-off' doctrine in the statistical learning theory. Since the generalization ability of the 'lazy trained' over-parametrized neural network can be well approximated by that of the neural tangent kernel regression, the curve of the excess risk (namely, the learning curve) of kernel ridge regression attracts increasing attention recently. However, most recent arguments on the learning curve are heuristic and are based on the 'Gaussian design' assumption. In this paper, under mild and more realistic assumptions, we rigorously provide a full characterization of the learning curve: elaborating the effect and the interplay of the choice of the regularization parameter, the source condition and the noise. In particular, our results suggest that the 'benign overfitting phenomenon' exists in very wide neural networks only when the noise level is small.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 21 of its 35 references that are in the map. First posted on arXiv 23 Sep 2023.

Builds on

21 of 35 references.

References · 21

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  4. 2019Benign Overfitting in Linear Regressioncited by 544
  5. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  6. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  7. 2018Gaussian Processes and Kernel Methods: A Review on Connections and Equivalencescited by 188
  8. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  9. 2017Sobolev Norm Learning Rates for Regularized Least-Squares Algorithmcited by 113
  10. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  11. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 83
  12. 2018Optimal Rates for Spectral Algorithms with Least-Squares Regression over Hilbert Spacescited by 59
9 more
  1. 2020Kernel Alignment Risk Estimator: Risk Prediction from Training Datacited by 54
  2. 2018Consistency of Interpolation with Laplace Kernels is a High-Dimensional Phenomenoncited by 44
  3. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  4. 2023Generalization Ability of Wide Neural Networks on ℝcited by 28
  5. 2023On the Optimality of Misspecified Spectral Algorithmscited by 28
  6. 2023On the Optimality of Misspecified Kernel Ridge Regressioncited by 20
  7. 2023Kernel interpolation generalizes poorlycited by 16
  8. 2021Learning curves for Gaussian process regression with power-law priors and targetscited by 14
  9. 2020Kernel Interpolation of High Dimensional Scattered Datacited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 6; 2026: 1.

19 papers.

Sort:

  1. 2023Generalization in Kernel Regression Under Realistic Assumptionscited by 26
  2. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  3. 2024Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrumcited by 13
  4. 2024Optimal Rates of Kernel Ridge Regression under Source Condition in Large Dimensionscited by 9
  5. 2024The phase diagram of kernel interpolation in large dimensionscited by 9
  6. 2025Models of Heavy-Tailed Mechanistic Universalitycited by 9
  7. 2024Generalization Error Curves for Analytic Spectral Algorithms under Power-law Decaycited by 8
  8. 2024A Comprehensive Analysis on the Learning Curve in Kernel Ridge Regressioncited by 6
  9. 2024Optimal Rates and Saturation for Noiseless Kernel Ridge Regressioncited by 3
  10. 2025Determinant Estimation under Memory Constraints and Neural Scaling Lawscited by 3
  11. 2024Improving Adaptivity via Over-Parameterization in Sequence Modelscited by 2
  12. 2025Beyond Benign Overfitting in Nadaraya-Watson Interpolatorscited by 2
7 more
  1. 2025On the Saturation Effects of Spectral Algorithms in Large Dimensionscited by 2
  2. 2024On the Pinsker bound of inner product kernel regression in large dimensionscited by 1
  3. 2024Benign overfitting in Fixed Dimension via Physics-Informed Learning with Smooth Inductive Biascited by 0
  4. 2024Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theoriescited by 0
  5. 2025Spectral Analysis of Molecular Features: When Richer Features Do Not Guarantee Better Generalizationcited by 0
  6. 2025A Special Case of Quadratic Extrapolation Under the Neural Tangent Kernelcited by 0
  7. 2026Large Dimensional Kernel Ridge Regression: Extending to Product Kernelscited by 0