mlsanityData to 7 Oct 2026

Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime

arXiv 2105.15004 ↗First posted 31 May 2021stat.ML cond-mat.dis-nn cs.LGcited by 95

Not on the map. Why?

Abstract (arXiv)

In this manuscript we consider Kernel Ridge Regression (KRR) under the Gaussian design. Exponents for the decay of the excess generalization error of KRR have been reported in various works under the assumption of power-law decay of eigenvalues of the features co-variance. These decays were, however, provided for sizeably different setups, namely in the noiseless case with constant regularization and in the noisy optimally regularized case. Intermediary settings have been left substantially uncharted. In this work, we unify and extend this line of work, providing characterization of all regimes and excess error decay rates that can be observed in terms of the interplay of noise and regularization. In particular, we show the existence of a transition in the noisy setting between the noiseless exponents to its noisy values as the sample complexity is increased. Finally, we illustrate how this crossover can also be observed on real data sets.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Builds on 30 of its 54 references that are in the map. First posted on arXiv 31 May 2021.

Builds on

30 of 54 references.

References · 30

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2018On Lazy Training in Differentiable Programmingcited by 671
  4. 2017Deep Neural Networks as Gaussian Processescited by 617
  5. 2019Benign Overfitting in Linear Regressioncited by 544
  6. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  7. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  8. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  9. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  10. 2019Two models of double descent for weak featurescited by 194
  11. 2018Gaussian Processes and Kernel Methods: A Review on Connections and Equivalencescited by 188
  12. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
18 more
  1. 2020Benign overfitting in ridge regressioncited by 152
  2. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  3. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  4. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  5. 2017Sobolev Norm Learning Rates for Regularized Least-Squares Algorithmcited by 113
  6. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  7. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  8. 2017A Random Matrix Approach to Neural Networkscited by 80
  9. 2018Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passescited by 69
  10. 2020A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descentcited by 69
  11. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 61
  12. 2018Optimal Rates for Spectral Algorithms with Least-Squares Regression over Hilbert Spacescited by 59
  13. 2020Kernel Alignment Risk Estimator: Risk Prediction from Training Datacited by 54
  14. 2018Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularizationcited by 52
  15. 2021Last iterate convergence of SGD for Least-Squares in the Interpolation regimecited by 41
  16. 2020Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Modelcited by 39
  17. 2020Kernel regression in high dimensions: Refined analysis beyond double descentcited by 39
  18. 2019Kernel Truncated Randomized Ridge Regression: Optimal Rates and Low Noise Accelerationcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 13; 2023: 16; 2024: 39; 2025: 15; 2026: 9.

95 papers.

Sort:

By area:

  1. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 143
  2. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 127
  3. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  4. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
  5. 2024Model Collapse Demystified: The Case of Regressioncited by 80
  6. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  7. 2024Strong Model Collapsecited by 62
  8. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  9. 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 50
  10. 2023High-dimensional analysis of double descent for linear regression with random projectionscited by 47
  11. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  12. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
38 more
  1. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  2. 2024A Simple Model of Inference Scaling Lawscited by 36
  3. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  4. 2023On the Optimality of Misspecified Spectral Algorithmscited by 28
  5. 2023Deep neural networks have an inbuilt Occam's razorcited by 28
  6. 2022Gaussian Universality of Perceptrons with Random Labelscited by 26
  7. 2023Generalization in Kernel Regression Under Realistic Assumptionscited by 26
  8. 2021The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networkscited by 25
  9. 2023Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimationcited by 25
  10. 2022Learning sparse features can lead to overfitting in neural networkscited by 23
  11. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  12. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  13. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  14. 2021Learning with convolution and pooling operations in kernel methodscited by 21
  15. 2024High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Lawscited by 21
  16. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  17. 2023On the Optimality of Misspecified Kernel Ridge Regressioncited by 20
  18. 2023Six Lectures on Linearized Neural Networkscited by 19
  19. 2023On the Asymptotic Learning Curves of Kernel Ridge Regression under Power-law Decaycited by 19
  20. 2025Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Lawcited by 19
  21. 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 18
  22. 2022Error Scaling Laws for Kernel Classification under Source and Capacity Conditionscited by 17
  23. 2023More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatorycited by 17
  24. 2022What Can Be Learnt With Wide Convolutional Neural Networks?cited by 16
  25. 2024Loss-to-Loss Prediction: Scaling Laws for All Datasetscited by 15
  26. 2025Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Timecited by 15
  27. 2021Learning curves for Gaussian process regression with power-law priors and targetscited by 14
  28. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 14
  29. 2024Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrumcited by 13
  30. 2024Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimensioncited by 13
  31. 2022Failure and success of the spectral bias prediction for Kernel Ridge Regression: the case of low-dimensional datacited by 12
  32. 2024High-dimensional learning of narrow neural networkscited by 12
  33. 2022A view of mini-batch SGD via generating functions: conditions of convergence, phase transitions, benefit from negative momentacited by 11
  34. 2022Tight Convergence Rate Bounds for Optimization Under Power Law Spectral Conditionscited by 9
  35. 2024Optimal Rates of Kernel Ridge Regression under Source Condition in Large Dimensionscited by 9
  36. 2024The phase diagram of kernel interpolation in large dimensionscited by 9
  37. 2022Kernel Methods and Multi-layer Perceptrons Learn Linear Models in High Dimensionscited by 8
  38. 2024Generalization Error Curves for Analytic Spectral Algorithms under Power-law Decaycited by 8

50 most cited of 95 All 95 (JSON)