mlsanityData to 14 Oct 2026

Characterizing the Spectrum of the NTK via a Power Series Expansion

arXiv 2211.07844 ↗First posted 15 Nov 2022cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Under mild conditions on the network initialization we derive a power series expansion for the Neural Tangent Kernel (NTK) of arbitrarily deep feedforward networks in the infinite width limit. We provide expressions for the coefficients of this power series which depend on both the Hermite coefficients of the activation function as well as the depth of the network. We observe faster decay of the Hermite coefficients leads to faster decay in the NTK coefficients and explore the role of depth. Using this series, first we relate the effective rank of the NTK to the effective rank of the input-data Gram. Second, for data drawn uniformly on the sphere we study the eigenvalues of the NTK, analyzing the impact of the choice of activation function. Finally, for generic data and activation functions with sufficiently fast Hermite coefficient decay, we derive an asymptotic upper bound on the spectrum of the NTK.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 49 of its 69 references that are in the map. First posted on arXiv 15 Nov 2022.

Builds on

49 of 69 references.

References · 49

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  6. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  7. 2017Deep Neural Networks as Gaussian Processescited by 622
  8. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  9. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  10. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  11. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  12. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
37 more
  1. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  2. 2016Deep Information Propagationcited by 195
  3. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  4. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  5. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
  6. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 159
  7. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
  8. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
  9. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  10. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  11. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 135
  12. 2015All you need is a good initcited by 124
  13. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 121
  14. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 96
  15. 2018Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approachcited by 91
  16. 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
  17. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 85
  18. 2017A Random Matrix Approach to Neural Networkscited by 80
  19. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 77
  20. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  21. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  22. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  23. 2019Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobiancited by 67
  24. 2020Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycited by 66
  25. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  26. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
  27. 2016Diverse Neural Network Learns True Target Functionscited by 33
  28. 2022Fast Neural Kernel Embeddings for General Activationscited by 18
  29. 2022On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernelscited by 16
  30. 2021Learning curves for Gaussian process regression with power-law priors and targetscited by 15
  31. 2019Effect of Activation Functions on the Training of Overparametrized Neural Netscited by 14
  32. 2020Dimensionality reduction, regularization, and generalization in overparameterized regressionscited by 13
  33. 2021Activation function design for deep networks: linearity and effective initialisationcited by 13
  34. 2021Reverse Engineering the Neural Tangent Kernelcited by 12
  35. 2022Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regimecited by 11
  36. 2022Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networkscited by 9
  37. 2022TorchNTK: A Library for Calculation of Neural Tangent Kernels of PyTorch Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 4; 2026: 10.

22 papers.

Sort:

  1. 2024Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimensioncited by 13
  2. 2023Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrumcited by 9
  3. 2024Near-Interpolators: Rapid Norm Growth and the Trade-Off between Interpolation and Generalizationcited by 8
  4. 2024NTK-Guided Few-Shot Class Incremental Learningcited by 7
  5. 2023Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networkscited by 6
  6. 2024Variational Search Distributionscited by 6
  7. 2026Richer Bayesian Last Layers with Subsampled NTK Featurescited by 4
  8. 2024Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspectivecited by 3
  9. 2025Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networkscited by 3
  10. 2025Fast Generalized Neural Tangent Kernel Statistics via Trace Estimationcited by 3
  11. 2025On the Complexity-Faithfulness Trade-off of Gradient-Based Explanationscited by 2
  12. 2023A note on regularised NTK dynamics with an application to PAC-Bayesian trainingcited by 1
10 more
  1. 2026On the Convergence Behavior of Preconditioned Gradient Descent Toward the Rich Learning Regimecited by 1
  2. 2026Curvature-Guided LoRA: Matching Full Fine-Tuning in Function Spacecited by 1
  3. 2025Revisiting the Neural Tangent Kernel: the role of large width and depthcited by 0
  4. 2026FlexAct: Why Learn when you can Pick?cited by 0
  5. 2026Pseudo-differential-enhanced physics-informed neural networkscited by 0
  6. 2026The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?cited by 0
  7. 2026Label-NTK Alignments and A Tighter Convergence Bound in the NTK Regimecited by 0
  8. 2026A Data-dependent Early Stopping Rule using Rademacher Complexity with L1-normcited by 0
  9. 2026Dimension-Free Rank Lifting from Random Hyperplane Arrangementscited by 0
  10. 2026Finite-Sample Approximation of Hessian-Guided Perturbed Wasserstein Gradient Flowscited by 0