mlsanityData to 7 Oct 2026

Learning curves for Gaussian process regression with power-law priors and targets

arXiv 2110.12231 ↗First posted 23 Oct 2021cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

We characterize the power-law asymptotics of learning curves for Gaussian process regression (GPR) under the assumption that the eigenspectrum of the prior and the eigenexpansion coefficients of the target function follow a power law. Under similar assumptions, we leverage the equivalence between GPR and kernel ridge regression (KRR) to show the generalization error of KRR. Infinitely wide neural networks can be related to GPR with respect to the neural network GP kernel and the neural tangent kernel, which in several cases is known to have a power-law spectrum. Hence our methods can be applied to study the generalization error of infinitely wide neural networks. We present toy experiments demonstrating the theory.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 33 of its 63 references that are in the map. First posted on arXiv 23 Oct 2021.

Builds on

33 of 63 references.

References · 33

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  3. 2017Deep Neural Networks as Gaussian Processescited by 617
  4. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  5. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  6. 2021Explaining Neural Scaling Lawscited by 353
  7. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  8. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  9. 2018Gaussian Processes and Kernel Methods: A Review on Connections and Equivalencescited by 188
  10. 2018To understand deep learning we need to understand kernel learningcited by 186
  11. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  12. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
21 more
  1. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  2. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  3. 2020On Information Gain and Regret Bounds in Gaussian Process Banditscited by 156
  4. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  5. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  6. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
  7. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  8. 2017Sobolev Norm Learning Rates for Regularized Least-Squares Algorithmcited by 113
  9. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  10. 2016Optimal Rates For Regularization Of Statistical Inverse Learning Problemscited by 68
  11. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  12. 2021The Shape of Learning Curves: a Reviewcited by 67
  13. 2020A Theory of Universal Learningcited by 59
  14. 2020Every Model Learned by Gradient Descent Is Approximately a Kernel Machinecited by 53
  15. 2020Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regimecited by 35
  16. 2019Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigmcited by 20
  17. 2018Online learning with kernel lossescited by 12
  18. 2019Kernel Truncated Randomized Ridge Regression: Optimal Rates and Low Noise Accelerationcited by 12
  19. 2019Minimizers of the Empirical Risk and Risk Monotonicitycited by 11
  20. 2018Relative concentration bounds for the spectrum of kernel matricescited by 4
  21. 2021Universal scaling laws in the gradient descent training of neural networkscited by 4

Built on by

Cited by year

Citing papers per year: 2022: 7; 2023: 2; 2024: 4; 2025: 1.

14 papers.

Sort:

  1. 2022Contrasting random and learned features in deep Bayesian linear regressioncited by 24
  2. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  3. 2023On the Optimality of Misspecified Kernel Ridge Regressioncited by 20
  4. 2023On the Asymptotic Learning Curves of Kernel Ridge Regression under Power-law Decaycited by 19
  5. 2022Error Scaling Laws for Kernel Classification under Source and Capacity Conditionscited by 17
  6. 2024Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimensioncited by 13
  7. 2022Failure and success of the spectral bias prediction for Kernel Ridge Regression: the case of low-dimensional datacited by 12
  8. 2022A view of mini-batch SGD via generating functions: conditions of convergence, phase transitions, benefit from negative momentacited by 11
  9. 2022Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processescited by 10
  10. 2022Tight Convergence Rate Bounds for Optimization Under Power Law Spectral Conditionscited by 9
  11. 2024Breaking Neural Network Scaling Laws with Modularitycited by 5
  12. 2025Uncertainty-Aware Diagnostics for Physics-Informed Machine Learningcited by 3
2 more
  1. 2024Generalization error of spectral algorithmscited by 1
  2. 2024On the Pinsker bound of inner product kernel regression in large dimensionscited by 1