Generalization error of spectral algorithms
Not on the map. Why?
Abstract (arXiv)
The asymptotically precise estimation of the generalization of kernel methods has recently received attention due to the parallels between neural networks and their associated kernels. However, prior works derive such estimates for training by kernel ridge regression (KRR), whereas neural networks are typically trained with gradient descent (GD). In the present work, we consider the training of kernels with a family of textitspectral algorithms specified by profile h(λ), and including KRR and GD as special cases. Then, we derive the generalization error as a functional of learning profile h(λ) for two data models: high-dimensional Gaussian and low-dimensional translation-invariant model. Under power-law assumptions on the spectrum of the kernel and target, we use our framework to (i) give full loss asymptotics for both noisy and noiseless observations (ii) show that the loss localizes on certain spectral scales, giving a new perspective on the KRR saturation phenomenon (iii) conjecture, and demonstrate for the considered data models, the universality of the loss w.r.t. non-spectral details of the problem, but only in case of noisy observation.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 19 of its 29 references that are in the map. First posted on arXiv 18 Mar 2024.
Builds on
19 of 29 references.
References · 19
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
- 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
- 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
- 2020Benign overfitting in ridge regressioncited by 152
- 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
- 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
- 2018Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passescited by 69
- 2018Optimal Rates for Spectral Algorithms with Least-Squares Regression over Hilbert Spacescited by 59
- 2020Implicit Regularization of Random Feature Modelscited by 54
7 more
- 2020Kernel Alignment Risk Estimator: Risk Prediction from Training Datacited by 54
- 2018Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularizationcited by 52
- 2021Fast Adaptation with Linearized Neural Networkscited by 32
- 2021Properties of the After Kernelcited by 27
- 2021Learning curves for Gaussian process regression with power-law priors and targetscited by 14
- 2022On the Inconsistency of Kernel Ridgeless Regression in Fixed Dimensionscited by 13
- 2016Optimal rates for the regularized learning algorithms under general source conditioncited by 11
Built on by
Citing papers per year: 2025: 1.
1 paper.