mlsanityData to 14 Oct 2026

Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regime

arXiv 2206.02927 ↗First posted 6 Jun 2022stat.ML cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

We provide quantitative bounds measuring the L2 difference in function space between the trajectory of a finite-width network trained on finitely many samples from the idealized kernel dynamics of infinite width and infinite data. An implication of the bounds is that the network is biased to learn the top eigenfunctions of the Neural Tangent Kernel not just on the training set but over the entire input space. This bias depends on the model architecture and input distribution alone and thus does not depend on the target function which does not need to be in the RKHS of the kernel. The result is valid for deep architectures with fully connected, convolutional, and residual layers. Furthermore the width does not need to grow polynomially with the number of samples in order to obtain high probability bounds up to a stopping time. The proof exploits the low-effective-rank property of the Fisher Information Matrix at initialization, which implies a low effective dimension of the model (far smaller than the number of parameters). We conclude that local capacity control from the low effective rank of the Fisher Information Matrix is still underexplored theoretically.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 44 of its 84 references that are in the map. First posted on arXiv 6 Jun 2022.

Builds on

44 of 84 references.

References · 44

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  4. 2018On the Spectral Bias of Neural Networkscited by 1,341
  5. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  6. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  7. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  8. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  9. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  10. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  11. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  12. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
32 more
  1. 2017Implicit Regularization in Matrix Factorizationcited by 322
  2. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  3. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  4. 2018Training behavior of deep neural network in frequency domaincited by 174
  5. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 174
  6. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 159
  7. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
  8. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  9. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 135
  10. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  11. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 121
  12. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 121
  13. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
  14. 2018Convergence of Gradient Descent on Separable Datacited by 110
  15. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  16. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  17. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  18. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 77
  19. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  20. 2019Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobiancited by 67
  21. 2020Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycited by 66
  22. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
  23. 2019Gradient Dynamics of Shallow Univariate ReLU Networkscited by 46
  24. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
  25. 2017Geometry of Optimization and Implicit Regularization in Deep Learningcited by 39
  26. 2020Deep Networks and the Multiple Manifold Problemcited by 35
  27. 2019A type of generalization error induced by initialization in deep neural networkscited by 33
  28. 2019Pathological spectra of the Fisher information metric and its variants in deep neural networkscited by 32
  29. 2019On Learning Over-parameterized Neural Networks: A Functional Approximation Perspectivecited by 28
  30. 2020On the exact computation of linear frequency principle dynamics and its generalizationcited by 16
  31. 2021Does the Data Induce Capacity Control in Deep Learning?cited by 14
  32. 2022Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networkscited by 9

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 4; 2024: 3; 2025: 2.

11 papers.

  1. 2023Generalization in Kernel Regression Under Realistic Assumptionscited by 27
  2. 2024A distributional simplicity bias in the learning dynamics of transformerscited by 24
  3. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  4. 2024Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimensioncited by 13
  5. 2022A view of mini-batch SGD via generating functions: conditions of convergence, phase transitions, benefit from negative momentacited by 11
  6. 2023Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrumcited by 9
  7. 2023On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspectivecited by 6
  8. 2024Train Faster, Perform Better: Modular Adaptive Training in Over-Parameterized Modelscited by 3
  9. 2025Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networkscited by 3
  10. 2025Characterising the Inductive Biases of Neural Networks on Boolean Datacited by 1
  11. 2023Optimization dependent generalization bound for ReLU networks based on sensitivity in the tangent bundlecited by 0