Demystifying Spectral Bias on Real-World Data
Not on the map. Why?
Abstract (arXiv)
Kernel ridge regression (KRR) and Gaussian processes (GPs) are fundamental tools in statistics and machine learning, with recent applications to highly over-parameterized deep neural networks. The ability of these tools to learn a target function is directly related to the eigenvalues of their kernel sampled on the input data distribution. Targets that have support on higher eigenvalues are more learnable. However, solving such eigenvalue problems on real-world data remains a challenge. Here, we consider cross-dataset learnability and show that one may use eigenvalues and eigenfunctions associated with highly idealized data measures to reveal spectral bias on complex datasets and bound learnability on real-world data. This allows us to leverage various symmetries that realistic kernels manifest to unravel their spectral bias.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 30 of its 64 references that are in the map. First posted on arXiv 4 Jun 2024.
Builds on
30 of 64 references.
References · 30
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2022In-context Learning and Induction Headscited by 930
- 2017Deep Neural Networks as Gaussian Processescited by 617
- 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
- 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
- 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
- 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
- 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
- 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
- 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
- 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
18 more
- 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
- 2024How Transformers Learn Causal Structure with Gradient Descentcited by 114
- 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
- 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 83
- 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 72
- 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
- 2021Out of Distribution Generalization in Machine Learningcited by 59
- 2020Predicting the outputs of finite deep neural networks trained with noisy gradientscited by 31
- 2021Approximation and Learning with Deep Convolutional Models: a Kernel Perspectivecited by 31
- 2023The Exact Sample Complexity Gain from Invariances for Kernel Regressioncited by 27
- 2021Out-of-Distribution Generalization in Kernel Regressioncited by 21
- 2021Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networkscited by 17
- 2022What Can Be Learnt With Wide Convolutional Neural Networks?cited by 16
- 2022On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernelscited by 15
- 2024A Short Survey on Importance Weighting for Machine Learningcited by 15
- 2021On the Sample Complexity of Learning under Invariance and Geometric Stabilitycited by 13
- 2024Towards Understanding Inductive Bias in Transformers: A View From Infinitycited by 13
- 2023Towards a Unified Analysis of Kernel-based Methods Under Covariate Shiftcited by 12
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.