# Characterizing the Spectrum of the NTK via a Power Series Expansion

arXiv: https://arxiv.org/abs/2211.07844 · HTML: https://mlsanity.com/paper/2211.07844/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 69 references that are in the map.
- First posted on arXiv 15 Nov 2022.

## Builds on (most cited first, 30 of 49)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks](https://mlsanity.com/paper/1901.08584/) (2019)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [Kernel and Rich Regimes in Overparametrized Models](https://mlsanity.com/paper/2002.09277/) (2020)
- [Exponential expressivity in deep neural networks through transient chaos](https://mlsanity.com/paper/1606.05340/) (2016)
- [Deep Information Propagation](https://mlsanity.com/paper/1611.01232/) (2016)
- [On the Inductive Bias of Neural Tangent Kernels](https://mlsanity.com/paper/1905.12173/) (2019)
- [Towards moderate overparameterization: global convergence guarantees for training shallow neural networks](https://mlsanity.com/paper/1902.04674/) (2019)
- [Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes](https://mlsanity.com/paper/1810.05148/) (2018)
- [Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks](https://mlsanity.com/paper/1903.11680/) (2019)
- [Finite Versus Infinite Neural Networks: an Empirical Study](https://mlsanity.com/paper/2007.15801/) (2020)
- [Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks](https://mlsanity.com/paper/1811.08888/) (2018)
- [Frequency Bias in Neural Networks for Input of Non-Uniform Density](https://mlsanity.com/paper/2003.04560/) (2020)
- [Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity](https://mlsanity.com/paper/1602.05897/) (2016)
- [The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies](https://mlsanity.com/paper/1906.00425/) (2019)
- [All you need is a good init](https://mlsanity.com/paper/1511.06422/) (2015)
- [An Improved Analysis of Training Over-parameterized Deep Neural Networks](https://mlsanity.com/paper/1906.04688/) (2019)
- [Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime](https://mlsanity.com/paper/2105.15004/) (2021)
- [Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach](https://mlsanity.com/paper/1806.01316/) (2018)
- [Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks](https://mlsanity.com/paper/2012.11654/) (2020)
- [Deep Equals Shallow for ReLU Networks in Kernel Regimes](https://mlsanity.com/paper/2009.14397/) (2020)
- [A Random Matrix Approach to Neural Networks](https://mlsanity.com/paper/1702.05419/) (2017)

## Built on by (most cited first, 22 of 22)

- [Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimension](https://mlsanity.com/paper/2405.14630/) (2024)
- [Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrum](https://mlsanity.com/paper/2307.14531/) (2023)
- [Near-Interpolators: Rapid Norm Growth and the Trade-Off between Interpolation and Generalization](https://mlsanity.com/paper/2403.07264/) (2024)
- [NTK-Guided Few-Shot Class Incremental Learning](https://mlsanity.com/paper/2403.12486/) (2024)
- [Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks](https://mlsanity.com/paper/2305.08813/) (2023)
- [Variational Search Distributions](https://mlsanity.com/paper/2409.06142/) (2024)
- [Richer Bayesian Last Layers with Subsampled NTK Features](https://mlsanity.com/paper/2602.01279/) (2026)
- [Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspective](https://mlsanity.com/paper/2407.17120/) (2024)
- [Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networks](https://mlsanity.com/paper/2506.22429/) (2025)
- [Fast Generalized Neural Tangent Kernel Statistics via Trace Estimation](https://mlsanity.com/paper/2511.10796/) (2025)
- [On the Complexity-Faithfulness Trade-off of Gradient-Based Explanations](https://mlsanity.com/paper/2508.10490/) (2025)
- [A note on regularised NTK dynamics with an application to PAC-Bayesian training](https://mlsanity.com/paper/2312.13259/) (2023)
- [On the Convergence Behavior of Preconditioned Gradient Descent Toward the Rich Learning Regime](https://mlsanity.com/paper/2601.03162/) (2026)
- [Curvature-Guided LoRA: Matching Full Fine-Tuning in Function Space](https://mlsanity.com/paper/2603.29824/) (2026)
- [Revisiting the Neural Tangent Kernel: the role of large width and depth](https://mlsanity.com/paper/2511.07272/) (2025)
- [FlexAct: Why Learn when you can Pick?](https://mlsanity.com/paper/2601.06441/) (2026)
- [Pseudo-differential-enhanced physics-informed neural networks](https://mlsanity.com/paper/2602.14663/) (2026)
- [The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?](https://mlsanity.com/paper/2605.20749/) (2026)
- [Label-NTK Alignments and A Tighter Convergence Bound in the NTK Regime](https://mlsanity.com/paper/2605.25275/) (2026)
- [A Data-dependent Early Stopping Rule using Rademacher Complexity with L1-norm](https://mlsanity.com/paper/2608.24210/) (2026)
- [Dimension-Free Rank Lifting from Random Hyperplane Arrangements](https://mlsanity.com/paper/2609.39855/) (2026)
- [Finite-Sample Approximation of Hessian-Guided Perturbed Wasserstein Gradient Flows](https://mlsanity.com/paper/2610.10218/) (2026)

## Abstract (arXiv)

Under mild conditions on the network initialization we derive a power series expansion for the Neural Tangent Kernel (NTK) of arbitrarily deep feedforward networks in the infinite width limit. We provide expressions for the coefficients of this power series which depend on both the Hermite coefficients of the activation function as well as the depth of the network. We observe faster decay of the Hermite coefficients leads to faster decay in the NTK coefficients and explore the role of depth. Using this series, first we relate the effective rank of the NTK to the effective rank of the input-data Gram. Second, for data drawn uniformly on the sphere we study the eigenvalues of the NTK, analyzing the impact of the choice of activation function. Finally, for generic data and activation functions with sufficiently fast Hermite coefficient decay, we derive an asymptotic upper bound on the spectrum of the NTK.