# Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm

arXiv: https://arxiv.org/abs/1905.10843 · HTML: https://mlsanity.com/paper/1905.10843/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 May 2019.

## Built on by (most cited first, 20 of 20)

- [A Solvable Model of Neural Scaling Laws](https://mlsanity.com/paper/2210.16859/) (2022)
- [Learning Curve Theory](https://mlsanity.com/paper/2102.04074/) (2021)
- [ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction](https://mlsanity.com/paper/2105.10446/) (2021)
- [Locality defeats the curse of dimensionality in convolutional teacher-student scenarios](https://mlsanity.com/paper/2106.08619/) (2021)
- [Probing transfer learning with a model of synthetic correlated datasets](https://mlsanity.com/paper/2106.05418/) (2021)
- [The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimes](https://mlsanity.com/paper/2212.12147/) (2022)
- [Contrasting random and learned features in deep Bayesian linear regression](https://mlsanity.com/paper/2203.00573/) (2022)
- [Learning curves for Gaussian process regression with power-law priors and targets](https://mlsanity.com/paper/2110.12231/) (2021)
- [Loss-to-Loss Prediction: Scaling Laws for All Datasets](https://mlsanity.com/paper/2411.12925/) (2024)
- [On the interplay between data structure and loss function in classification problems](https://mlsanity.com/paper/2103.05524/) (2021)
- [Why flatness does and does not correlate with generalization for deep neural networks](https://mlsanity.com/paper/2103.06219/) (2021)
- [Failure and success of the spectral bias prediction for Kernel Ridge Regression: the case of low-dimensional data](https://mlsanity.com/paper/2202.03348/) (2022)
- [A Theoretical-Empirical Approach to Estimating Sample Complexity of DNNs](https://mlsanity.com/paper/2105.01867/) (2021)
- [Turing-Universal Learners with Optimal Scaling Laws](https://mlsanity.com/paper/2111.05321/) (2021)
- [Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling](https://mlsanity.com/paper/2512.19905/) (2025)
- [Jailbreak Scaling Laws for Large Language Models: Polynomial-Exponential Crossover](https://mlsanity.com/paper/2603.11331/) (2026)
- [Feature learning is decoupled from generalization in high capacity neural networks](https://mlsanity.com/paper/2507.19680/) (2025)
- [Investigating classification learning curves for automatically generated and labelled plant images](https://mlsanity.com/paper/2205.10955/) (2022)
- [Sparsity-depth Tradeoff in Infinitely Wide Deep Neural Networks](https://mlsanity.com/paper/2305.10550/) (2023)
- [Position: Many generalization measures for deep learning are fragile](https://mlsanity.com/paper/2510.18934/) (2025)

## Abstract (arXiv)

How many training data are needed to learn a supervised task? It is often observed that the generalization error decreases as n-β where n is the number of training examples and β an exponent that depends on both data and algorithm. In this work we measure β when applying kernel methods to real datasets. For MNIST we find β≈ 0.4 and for CIFAR10 β≈ 0.1, for both regression and classification tasks, and for Gaussian or Laplace kernels. To rationalize the existence of non-trivial exponents that can be independent of the specific kernel used, we study the Teacher-Student framework for kernels. In this scheme, a Teacher generates data according to a Gaussian random field, and a Student learns them via kernel regression. With a simplifying assumption -- namely that the data are sampled from a regular lattice -- we derive analytically β for translation invariant kernels, using previous results from the kriging literature. Provided that the Student is not too sensitive to high frequencies, β depends only on the smoothness and dimension of the training data. We confirm numerically that these predictions hold when the training points are sampled at random on a hypersphere. Overall, the test error is found to be controlled by the magnitude of the projection of the true function on the kernel eigenvectors whose rank is larger than n. Using this idea we predict relate the exponent β to an exponent a describing how the coefficients of the true function in the eigenbasis of the kernel decay with rank. We extract a from real data by performing kernel PCA, leading to β≈0.36 for MNIST and β≈0.07 for CIFAR10, in good agreement with observations. We argue that these rather large exponents are possible due to the small effective dimension of the data.