mlsanityData to 7 Oct 2026

On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernels

arXiv 2203.09255 ↗First posted 17 Mar 2022cs.LG cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

We study the properties of various over-parametrized convolutional neural architectures through their respective Gaussian process and neural tangent kernels. We prove that, with normalized multi-channel input and ReLU activation, the eigenfunctions of these kernels with the uniform measure are formed by products of spherical harmonics, defined over the channels of the different pixels. We next use hierarchical factorizable kernels to bound their respective eigenvalues. We show that the eigenvalues decay polynomially, quantify the rate of decay, and derive measures that reflect the composition of hierarchical features in these networks. Our results provide concrete quantitative characterization of over-parameterized convolutional network architectures.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 30 of its 56 references that are in the map. First posted on arXiv 17 Mar 2022.

Builds on

30 of 56 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2016Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Networkcited by 1,898
  5. 2017Deep Image Priorcited by 1,097
  6. 2015Accurate Image Super-Resolution Using Very Deep Convolutional Networkscited by 902
  7. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  8. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  9. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  10. 2017Understanding the Effective Receptive Field in Deep Convolutional Neural Networkscited by 549
  11. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  12. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
18 more
  1. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  2. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  3. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  4. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  5. 2015On the Expressive Power of Deep Learning: A Tensor Analysiscited by 97
  6. 2021Learning with invariances in random features and kernel modelscited by 89
  7. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 83
  8. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  9. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  10. 2014Convolutional Kernel Networkscited by 47
  11. 2019Denoising and Regularization via Exploiting the Structural Bias of Convolutional Generatorscited by 40
  12. 2021Approximation and Learning with Deep Convolutional Models: a Kernel Perspectivecited by 31
  13. 2016End-to-End Kernel Learning with Supervised Convolutional Kernel Networkscited by 28
  14. 2021Locality defeats the curse of dimensionality in convolutional teacher-student scenarioscited by 26
  15. 2021Learning with convolution and pooling operations in kernel methodscited by 21
  16. 2021Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networkscited by 17
  17. 2020The Neural Tangent Link Between CNN Denoisers and Non-Local Filterscited by 9
  18. 2020Harmonic Decompositions of Convolutional Networkscited by 6

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 3; 2024: 3; 2025: 6.

15 papers.

Sort:

  1. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  2. 2023Generalization in Kernel Regression Under Realistic Assumptionscited by 26
  3. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  4. 2022What Can Be Learnt With Wide Convolutional Neural Networks?cited by 16
  5. 2024Towards Understanding Inductive Bias in Transformers: A View From Infinitycited by 13
  6. 2022A Kernel Perspective of Skip Connections in Convolutional Networkscited by 12
  7. 2023Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrumcited by 9
  8. 2025Spectral Bias in Variational Quantum Machine Learningcited by 5
  9. 2024Understanding the dynamics of the frequency bias in neural networkscited by 4
  10. 2024Demystifying Spectral Bias on Real-World Datacited by 3
  11. 2025Neural Tangent Kernel of Neural Networks with Loss Informed by Differential Operatorscited by 3
  12. 2025Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networkscited by 3
3 more
  1. 2025General Fourier Feature Physics-Informed Extreme Learning Machine (GFF-PIELM) for High-Frequency PDEscited by 2
  2. 2025Characterising the Inductive Biases of Neural Networks on Boolean Datacited by 1
  3. 2025Feature learning is decoupled from generalization in high capacity neural networkscited by 1