# A non-asymptotic theory of Kernel Ridge Regression: deterministic equivalents, test error, and GCV estimator

arXiv: https://arxiv.org/abs/2403.08938 · HTML: https://mlsanity.com/paper/2403.08938/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2024.

## Built on by (most cited first, 28 of 28)

- [Scaling and renormalization in high-dimensional regression](https://mlsanity.com/paper/2405.00592/) (2024)
- [Dimension-free deterministic equivalents and scaling laws for random feature regression](https://mlsanity.com/paper/2405.15699/) (2024)
- [Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime](https://mlsanity.com/paper/2509.24882/) (2025)
- [Weak-to-Strong Generalization Even in Random Feature Networks, Provably](https://mlsanity.com/paper/2503.02877/) (2025)
- [Risk and cross validation in ridge regression with correlated samples](https://mlsanity.com/paper/2408.04607/) (2024)
- [Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionality](https://mlsanity.com/paper/2409.03891/) (2024)
- [A Comprehensive Analysis on the Learning Curve in Kernel Ridge Regression](https://mlsanity.com/paper/2410.17796/) (2024)
- [Two-Point Deterministic Equivalence for Stochastic Gradient Dynamics in Linear Models](https://mlsanity.com/paper/2502.05074/) (2025)
- [Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws](https://mlsanity.com/paper/2509.24914/) (2025)
- [A general technique for approximating high-dimensional empirical kernel matrices](https://mlsanity.com/paper/2511.03892/) (2025)
- [Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling](https://mlsanity.com/paper/2512.19905/) (2025)
- [Understanding and Enhancing Mask-Based Pretraining towards Universal Representations](https://mlsanity.com/paper/2509.21650/) (2025)
- [Generalization, memorization, and overfitting for diffusion models trained in the lazy high-dimensional regime](https://mlsanity.com/paper/2608.23938/) (2026)
- [On the Pinsker bound of inner product kernel regression in large dimensions](https://mlsanity.com/paper/2409.00915/) (2024)
- [Beyond Real Data: Synthetic Data through the Lens of Regularization](https://mlsanity.com/paper/2510.08095/) (2025)
- [Breaking the curse of dimensionality for linear rules: optimal predictors over the ellipsoid](https://mlsanity.com/paper/2509.21174/) (2025)
- [Statistical Mechanics of Support Vector Regression](https://mlsanity.com/paper/2412.05439/) (2024)
- [Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theories](https://mlsanity.com/paper/2412.18756/) (2024)
- [Training NTK to Generalize with KARE](https://mlsanity.com/paper/2505.11347/) (2025)
- [A Random Matrix Theory of Masked Self-Supervised Regression](https://mlsanity.com/paper/2601.23208/) (2026)
- [Learning Curves and Benign Overfitting of Spectral Algorithms in Large Dimensions](https://mlsanity.com/paper/2604.23212/) (2026)
- [Large Dimensional Kernel Ridge Regression: Extending to Product Kernels](https://mlsanity.com/paper/2605.14524/) (2026)
- [Eigen-Spike Emergence and Quadratic Equivalents for Conjugate Kernels on Nonlinearly Separable Data](https://mlsanity.com/paper/2605.29669/) (2026)
- [How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks](https://mlsanity.com/paper/2606.28242/) (2026)
- [Learning between the peaks: sharp asymptotics for kernel ridge regression under power-law anisotropy](https://mlsanity.com/paper/2608.28564/) (2026)
- [When do data mixtures improve scaling laws? Insights from high-dimensional regression](https://mlsanity.com/paper/2609.38011/) (2026)
- [From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes](https://mlsanity.com/paper/2609.40148/) (2026)
- [Quadratic Weak-to-Strong Generalization in Random Feature Networks via Random Matrix Theory](https://mlsanity.com/paper/2610.09044/) (2026)

## Abstract (arXiv)

We consider learning an unknown target function f_* using kernel ridge regression (KRR) given i.i.d. data (ui,yi), i≤ n, where ui ∈ U is a covariate vector and yi = f_* (ui) +varepsiloni ∈ ℝ. A recent string of work has empirically shown that the test error of KRR can be well approximated by a closed-form estimate derived from an `equivalent' sequence model that only depends on the spectrum of the kernel operator. However, a theoretical justification for this equivalence has so far relied either on restrictive assumptions -- such as subgaussian independent eigenfunctions -- , or asymptotic derivations for specific kernels in high dimensions. In this paper, we prove that this equivalence holds for a general class of problems satisfying some spectral and concentration properties on the kernel eigendecomposition. Specifically, we establish in this setting a non-asymptotic deterministic approximation for the test error of KRR -- with explicit non-asymptotic bounds -- that only depends on the eigenvalues and the target function alignment to the eigenvectors of the kernel. Our proofs rely on a careful derivation of deterministic equivalents for random matrix functionals in the dimension free regime pioneered by Cheng and Montanari (2022). We apply this setting to several classical examples and show an excellent agreement between theoretical predictions and numerical simulations. These results rely on having access to the eigendecomposition of the kernel operator. Alternatively, we prove that, under this same setting, the generalized cross-validation (GCV) estimator concentrates on the test error uniformly over a range of ridge regularization parameter that includes zero (the interpolating solution). As a consequence, the GCV estimator can be used to estimate from data the test error and optimal regularization parameter for KRR.