# On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernels

arXiv: https://arxiv.org/abs/2203.09255 · HTML: https://mlsanity.com/paper/2203.09255/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 56 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Accurate Image Super-Resolution Using Very Deep Convolutional Networks](https://mlsanity.com/paper/1511.04587/) (2015)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [Understanding the Effective Receptive Field in Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.04128/) (2017)
- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [On the Inductive Bias of Neural Tangent Kernels](https://mlsanity.com/paper/1905.12173/) (2019)
- [Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes](https://mlsanity.com/paper/1810.05148/) (2018)
- [Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Review](https://mlsanity.com/paper/1611.00740/) (2016)
- [Frequency Bias in Neural Networks for Input of Non-Uniform Density](https://mlsanity.com/paper/2003.04560/) (2020)
- [The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies](https://mlsanity.com/paper/1906.00425/) (2019)
- [On the Expressive Power of Deep Learning: A Tensor Analysis](https://mlsanity.com/paper/1509.05009/) (2015)
- [Learning with invariances in random features and kernel models](https://mlsanity.com/paper/2102.13219/) (2021)
- [Deep Equals Shallow for ReLU Networks in Kernel Regimes](https://mlsanity.com/paper/2009.14397/) (2020)
- [On the Similarity between the Laplace and Neural Tangent Kernels](https://mlsanity.com/paper/2007.01580/) (2020)
- [Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHS](https://mlsanity.com/paper/2009.10683/) (2020)
- [Convolutional Kernel Networks](https://mlsanity.com/paper/1406.3332/) (2014)
- [Denoising and Regularization via Exploiting the Structural Bias of Convolutional Generators](https://mlsanity.com/paper/1910.14634/) (2019)
- [Approximation and Learning with Deep Convolutional Models: a Kernel Perspective](https://mlsanity.com/paper/2102.10032/) (2021)
- [End-to-End Kernel Learning with Supervised Convolutional Kernel Networks](https://mlsanity.com/paper/1605.06265/) (2016)
- [Locality defeats the curse of dimensionality in convolutional teacher-student scenarios](https://mlsanity.com/paper/2106.08619/) (2021)
- [Learning with convolution and pooling operations in kernel methods](https://mlsanity.com/paper/2111.08308/) (2021)
- [Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networks](https://mlsanity.com/paper/2112.05611/) (2021)
- [The Neural Tangent Link Between CNN Denoisers and Non-Local Filters](https://mlsanity.com/paper/2006.02379/) (2020)
- [Harmonic Decompositions of Convolutional Networks](https://mlsanity.com/paper/2003.12756/) (2020)

## Built on by (most cited first, 16 of 16)

- [Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models](https://mlsanity.com/paper/2305.12827/) (2023)
- [Generalization in Kernel Regression Under Realistic Assumptions](https://mlsanity.com/paper/2312.15995/) (2023)
- [Characterizing the Spectrum of the NTK via a Power Series Expansion](https://mlsanity.com/paper/2211.07844/) (2022)
- [What Can Be Learnt With Wide Convolutional Neural Networks?](https://mlsanity.com/paper/2208.01003/) (2022)
- [Towards Understanding Inductive Bias in Transformers: A View From Infinity](https://mlsanity.com/paper/2402.05173/) (2024)
- [A Kernel Perspective of Skip Connections in Convolutional Networks](https://mlsanity.com/paper/2211.14810/) (2022)
- [Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrum](https://mlsanity.com/paper/2307.14531/) (2023)
- [Spectral Bias in Variational Quantum Machine Learning](https://mlsanity.com/paper/2506.22555/) (2025)
- [Understanding the dynamics of the frequency bias in neural networks](https://mlsanity.com/paper/2405.14957/) (2024)
- [Neural Tangent Kernel of Neural Networks with Loss Informed by Differential Operators](https://mlsanity.com/paper/2503.11029/) (2025)
- [Demystifying Spectral Bias on Real-World Data](https://mlsanity.com/paper/2406.02663/) (2024)
- [Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networks](https://mlsanity.com/paper/2506.22429/) (2025)
- [General Fourier Feature Physics-Informed Extreme Learning Machine (GFF-PIELM) for High-Frequency PDEs](https://mlsanity.com/paper/2510.12293/) (2025)
- [Characterising the Inductive Biases of Neural Networks on Boolean Data](https://mlsanity.com/paper/2505.24060/) (2025)
- [Feature learning is decoupled from generalization in high capacity neural networks](https://mlsanity.com/paper/2507.19680/) (2025)
- [Operator-informed initialization for Fourier features physics-informed neural networks](https://mlsanity.com/paper/2610.03378/) (2026)

## Abstract (arXiv)

We study the properties of various over-parametrized convolutional neural architectures through their respective Gaussian process and neural tangent kernels. We prove that, with normalized multi-channel input and ReLU activation, the eigenfunctions of these kernels with the uniform measure are formed by products of spherical harmonics, defined over the channels of the different pixels. We next use hierarchical factorizable kernels to bound their respective eigenvalues. We show that the eigenvalues decay polynomially, quantify the rate of decay, and derive measures that reflect the composition of hierarchical features in these networks. Our results provide concrete quantitative characterization of over-parameterized convolutional network architectures.