mlsanityData to 7 Oct 2026

Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks

arXiv 2005.11879 ↗First posted 25 May 2020stat.ML cs.LG math.PRcited by 76

Not on the map. Why?

Abstract (arXiv)

We study the eigenvalue distributions of the Conjugate Kernel and Neural Tangent Kernel associated to multi-layer feedforward neural networks. In an asymptotic regime where network width is increasing linearly in sample size, under random initialization of the weights, and for input samples satisfying a notion of approximate pairwise orthogonality, we show that the eigenvalue distributions of the CK and NTK converge to deterministic limits. The limit for the CK is described by iterating the Marcenko-Pastur map across the hidden layers. The limit for the NTK is equivalent to that of a linear combination of the CK matrices across layers, and may be described by recursive fixed-point equations that extend this Marcenko-Pastur map. We demonstrate the agreement of these asymptotic predictions with the observed spectra for both synthetic and CIFAR-10 training data, and we perform a small simulation to investigate the evolutions of these spectra over training.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 14; 2023: 14; 2024: 16; 2025: 12; 2026: 8.

76 papers.

Sort:

  1. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  2. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
  3. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  4. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  5. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  6. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  7. 2023On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domainscited by 30
  8. 2023Generalization in Kernel Regression Under Realistic Assumptionscited by 26
  9. 2024Nonlinear spiked covariance matrices and signal propagation in deep neural networkscited by 25
  10. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  11. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  12. 2023Precise Asymptotic Analysis of Deep Random Feature Modelscited by 22
38 more
  1. 2021A Neural Tangent Kernel Perspective of GANscited by 21
  2. 2021Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networkscited by 21
  3. 2021Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networkscited by 20
  4. 2023Six Lectures on Linearized Neural Networkscited by 19
  5. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 14
  6. 2023Beyond the Universal Law of Robustness: Sharper Laws for Random Features and Neural Tangent Kernelscited by 13
  7. 2024Asymptotics of Learning with Deep Structured (Random) Featurescited by 13
  8. 2024Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimensioncited by 13
  9. 2022Largest Eigenvalues of the Conjugate Kernel of Single-Layered Neural Networkscited by 12
  10. 2023Fundamental limits of overparametrized shallow neural networks for supervised learningcited by 12
  11. 2024The Positivity of the Neural Tangent Kernelcited by 12
  12. 2022Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regimecited by 11
  13. 2023How Spurious Features Are Memorized: Precise Analysis for Random and NTK Featurescited by 11
  14. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11
  15. 2022Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processescited by 10
  16. 2023How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient Descentcited by 10
  17. 2021Analysis of One-Hidden-Layer Neural Networks via the Resolvent Methodcited by 9
  18. 2024From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networkscited by 9
  19. 2025Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methodscited by 9
  20. 2022Concentration of Random Feature Matrices in High-Dimensionscited by 8
  21. 2024BWS: Best Window Selection Based on Sample Scores for Data Pruning across Broad Rangescited by 8
  22. 2024SHAP values via sparse Fourier representationcited by 8
  23. 2025Kernel ridge regression under power-law data: spectrum and generalizationcited by 8
  24. 2021Random matrices in service of ML footprint: ternary random features with no performance losscited by 7
  25. 2022On skip connections and normalisation layers in deep optimisationcited by 7
  26. 2023Efficient kernel surrogates for neural network-based regressioncited by 7
  27. 2021Lower Bounds on the Generalization Error of Nonlinear Learning Modelscited by 6
  28. 2021Random Features for the Neural Tangent Kernelcited by 6
  29. 2022Overparameterized random feature regression with nearly orthogonal datacited by 6
  30. 2023Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networkscited by 6
  31. 2024Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixturescited by 6
  32. 2021Double-descent curves in neural networks: a new perspective using Gaussian processescited by 5
  33. 2021Low Dimensional Landscape Hypothesis is True: DNNs can be Trained in Tiny Subspacescited by 5
  34. 2021Node Feature Kernels Increase Graph Convolutional Network Robustnesscited by 5
  35. 2023Deterministic equivalent of the Conjugate Kernel matrix associated to Artificial Neural Networkscited by 5
  36. 2021Universal scaling laws in the gradient descent training of neural networkscited by 4
  37. 2022Behind the Scenes of Gradient Descent: A Trajectory Analysis via Basis Function Decompositioncited by 4
  38. 2025Low Rank Gradients and Where to Find Themcited by 4

50 most cited of 76 All 76 (JSON)