Data to 7 Oct 2026

Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks

arXiv 2002.02561 ↗First posted 7 Feb 2020cs.LG stat.MLcited by 200

Not on the map. Why?

Abstract (arXiv)

We derive analytical expressions for the generalization performance of kernel regression as a function of the number of training samples using theoretical methods from Gaussian processes and statistical physics. Our expressions apply to wide neural networks due to an equivalence between training them and kernel regression with the Neural Tangent Kernel (NTK). By computing the decomposition of the total generalization error due to different spectral components of the kernel, we identify a new spectral principle: as the size of the training set grows, kernel machines and neural networks fit successively higher spectral modes of the target function. When data are sampled from a uniform distribution on a high-dimensional hypersphere, dot product kernels, including NTK, exhibit learning stages where different frequency modes of the target function are learned. We verify our theory with simulations on synthetic data and MNIST dataset.

Key facts. Cited by 200 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 25; 2022: 26; 2023: 29; 2024: 49; 2025: 33; 2026: 37.

200 papers.

Sort:

By area:

2021Explaining Neural Scaling Lawscited by 353
2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
2023The Quantization Model of Neural Scalingcited by 137
2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 127
2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
2024A Dynamical Model of Neural Scaling Lawscited by 105
2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
2022A Solvable Model of Neural Scaling Lawscited by 101
2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95

38 more

2024Model Collapse Demystified: The Case of Regressioncited by 80
2021Learning Curve Theorycited by 79
2022Overview frequency principle/spectral bias in deep learningcited by 68
2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 52
2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 50
2024Scaling and renormalization in high-dimensional regressioncited by 45
2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
2023Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzlecited by 36
2024A Simple Model of Inference Scaling Lawscited by 36
2022Limitations of the NTK for Understanding Generalization in Deep Learningcited by 34
2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
2022Bandwidth Enables Generalization in Quantum Kernel Modelscited by 32
2025Superposition Yields Robust Neural Scalingcited by 30
2023Generalization Ability of Wide Neural Networks on ℝcited by 28
2023On the Optimality of Misspecified Spectral Algorithmscited by 28
2023Deep neural networks have an inbuilt Occam's razorcited by 28
2022An Equivalence Principle for the Spectrum of Random Inner-Product Kernel Matrices with Polynomial Scalingscited by 27
2021Locality defeats the curse of dimensionality in convolutional teacher-student scenarioscited by 26
2022Gaussian Universality of Perceptrons with Random Labelscited by 26
2023Generalization in Kernel Regression Under Realistic Assumptionscited by 26
2021A Theory of Neural Tangent Kernel Alignment and Its Influence on Trainingcited by 25
2021The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networkscited by 25
2022The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimescited by 25
2023Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimationcited by 25
2024Nonlinear spiked covariance matrices and signal propagation in deep neural networkscited by 25
2022Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experimentscited by 23
2022Learning sparse features can lead to overfitting in neural networkscited by 23
2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
2025Generalization through variance: how noise shapes inductive biases in diffusion modelscited by 23
2022Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regressioncited by 22
2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
2021Out-of-Distribution Generalization in Kernel Regressioncited by 21
2021Uniform Generalization Bounds for Overparameterized Neural Networkscited by 21

50 most cited of 200 All 200 (JSON)