# Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks

arXiv: https://arxiv.org/abs/2005.11879 · HTML: https://mlsanity.com/paper/2005.11879/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 May 2020.

## Built on by (most cited first, 30 of 77)

- [High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation](https://mlsanity.com/paper/2205.01445/) (2022)
- [Random Neural Networks in the Infinite Width Limit as Gaussian Processes](https://mlsanity.com/paper/2107.01562/) (2021)
- [Bayes-optimal Learning of Deep Random Networks of Extensive-width](https://mlsanity.com/paper/2302.00375/) (2023)
- [Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterization](https://mlsanity.com/paper/2205.10217/) (2022)
- [Deterministic equivalent and error universality of deep random features learning](https://mlsanity.com/paper/2302.00401/) (2023)
- [On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domains](https://mlsanity.com/paper/2305.02657/) (2023)
- [Spectral Evolution and Invariance in Linear-width Neural Networks](https://mlsanity.com/paper/2211.06506/) (2022)
- [Generalization in Kernel Regression Under Realistic Assumptions](https://mlsanity.com/paper/2312.15995/) (2023)
- [Nonlinear spiked covariance matrices and signal propagation in deep neural networks](https://mlsanity.com/paper/2402.10127/) (2024)
- [Dimension-free deterministic equivalents and scaling laws for random feature regression](https://mlsanity.com/paper/2405.15699/) (2024)
- [Characterizing the Spectrum of the NTK via a Power Series Expansion](https://mlsanity.com/paper/2211.07844/) (2022)
- [Precise Asymptotic Analysis of Deep Random Feature Models](https://mlsanity.com/paper/2302.06210/) (2023)
- [Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks](https://mlsanity.com/paper/2104.03093/) (2021)
- [A Neural Tangent Kernel Perspective of GANs](https://mlsanity.com/paper/2106.05566/) (2021)
- [Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks](https://mlsanity.com/paper/2109.09304/) (2021)
- [Six Lectures on Linearized Neural Networks](https://mlsanity.com/paper/2308.13431/) (2023)
- [A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilities](https://mlsanity.com/paper/2410.18938/) (2024)
- [Largest Eigenvalues of the Conjugate Kernel of Single-Layered Neural Networks](https://mlsanity.com/paper/2201.04753/) (2022)
- [Beyond the Universal Law of Robustness: Sharper Laws for Random Features and Neural Tangent Kernels](https://mlsanity.com/paper/2302.01629/) (2023)
- [Fundamental limits of overparametrized shallow neural networks for supervised learning](https://mlsanity.com/paper/2307.05635/) (2023)
- [Asymptotics of Learning with Deep Structured (Random) Features](https://mlsanity.com/paper/2402.13999/) (2024)
- [The Positivity of the Neural Tangent Kernel](https://mlsanity.com/paper/2404.12928/) (2024)
- [Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimension](https://mlsanity.com/paper/2405.14630/) (2024)
- [Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regime](https://mlsanity.com/paper/2206.02927/) (2022)
- [Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes](https://mlsanity.com/paper/2210.07612/) (2022)
- [How Spurious Features Are Memorized: Precise Analysis for Random and NTK Features](https://mlsanity.com/paper/2305.12100/) (2023)
- [Unifying Low Dimensional Spectra in Deep Learning](https://mlsanity.com/paper/2404.06106/) (2024)
- [How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient Descent](https://mlsanity.com/paper/2309.08044/) (2023)
- [Kernel ridge regression under power-law data: spectrum and generalization](https://mlsanity.com/paper/2510.04780/) (2025)
- [Analysis of One-Hidden-Layer Neural Networks via the Resolvent Method](https://mlsanity.com/paper/2105.05115/) (2021)

## Abstract (arXiv)

We study the eigenvalue distributions of the Conjugate Kernel and Neural Tangent Kernel associated to multi-layer feedforward neural networks. In an asymptotic regime where network width is increasing linearly in sample size, under random initialization of the weights, and for input samples satisfying a notion of approximate pairwise orthogonality, we show that the eigenvalue distributions of the CK and NTK converge to deterministic limits. The limit for the CK is described by iterating the Marcenko-Pastur map across the hidden layers. The limit for the NTK is equivalent to that of a linear combination of the CK matrices across layers, and may be described by recursive fixed-point equations that extend this Marcenko-Pastur map. We demonstrate the agreement of these asymptotic predictions with the observed spectra for both synthetic and CIFAR-10 training data, and we perform a small simulation to investigate the evolutions of these spectra over training.