mlsanityData to 13 Oct 2026

Sharp Asymptotics of Kernel Ridge Regression Beyond the Linear Regime

arXiv 2205.06798 ↗First posted 13 May 2022cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

The generalization performance of kernel ridge regression (KRR) exhibits a multi-phased pattern that crucially depends on the scaling relationship between the sample size n and the underlying dimension d. This phenomenon is due to the fact that KRR sequentially learns functions of increasing complexity as the sample size increases; when dk-1ll nll dk, only polynomials with degree less than k are learned. In this paper, we present sharp asymptotic characterization of the performance of KRR at the critical transition regions with n asymp dk, for k∈mathbbZ+. Our asymptotic characterization provides a precise picture of the whole learning process and clarifies the impact of various parameters (including the choice of the kernel function) on the generalization performance. In particular, we show that the learning curves of KRR can have a delicate "double descent" behavior due to specific bias-variance trade-offs at different polynomial scaling regimes.

Key facts. Cited by 18 papers in the map (data through 13 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 13 May 2022.

Builds on

20 of 36 references.

References · 20

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,610
  2. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 670
  3. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 573
  4. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 510
  5. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 411
  6. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  7. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  8. 2018To understand deep learning we need to understand kernel learningcited by 186
  9. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  10. 2019Linearized two-layers neural networks in high dimensioncited by 155
  11. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 147
  12. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
8 more
  1. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  2. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  3. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 98
  4. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  5. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 62
  6. 2018Consistency of Interpolation with Laplace Kernels is a High-Dimensional Phenomenoncited by 44
  7. 2022Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regressioncited by 43
  8. 2022An Equivalence Principle for the Spectrum of Random Inner-Product Kernel Matrices with Polynomial Scalingscited by 27

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 6; 2024: 7; 2025: 2.

18 papers.

Sort:

  1. 2024Scaling and renormalization in high-dimensional regressioncited by 46
  2. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  3. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  4. 2022Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regressioncited by 22
  5. 2024Asymptotics of Random Feature Regression Beyond the Linear Scaling Regimecited by 22
  6. 2023Six Lectures on Linearized Neural Networkscited by 19
  7. 2024Optimal Rates of Kernel Ridge Regression under Source Condition in Large Dimensionscited by 9
  8. 2024The phase diagram of kernel interpolation in large dimensionscited by 9
  9. 2022Overparameterized random feature regression with nearly orthogonal datacited by 6
  10. 2024Random features and polynomial rulescited by 6
  11. 2022Gradient flow in the gaussian covariate model: exact solution of learning curves and multiple descent structurescited by 5
  12. 2023Learning from higher-order statistics, efficiently: hypothesis tests, random features, and neural networkscited by 3
6 more
  1. 2025A general technique for approximating high-dimensional empirical kernel matricescited by 3
  2. 2025On the Saturation Effects of Spectral Algorithms in Large Dimensionscited by 2
  3. 2023Asymptotics of Bayesian Uncertainty Estimation in Random Features Regressioncited by 1
  4. 2023Flat Minima in Linear Estimation and an Extended Gauss Markov Theoremcited by 1
  5. 2024On the Pinsker bound of inner product kernel regression in large dimensionscited by 1
  6. 2024Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theoriescited by 0