mlsanityData to 14 Oct 2026

Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initialization

arXiv 2202.00553 ↗First posted 1 Feb 2022cs.LG cs.AIcited by 30

Not on the map. Why?

Abstract (arXiv)

Neural Tangent Kernel (NTK) is widely used to analyze overparametrized neural networks due to the famous result by Jacot et al. (2018): in the infinite-width limit, the NTK is deterministic and constant during training. However, this result cannot explain the behavior of deep networks, since it generally does not hold if depth and width tend to infinity simultaneously. In this paper, we study the NTK of fully-connected ReLU networks with depth comparable to width. We prove that the NTK properties depend significantly on the depth-to-width ratio and the distribution of parameters at initialization. In fact, our results indicate the importance of the three phases in the hyperparameter space identified in Poole et al. (2016): ordered, chaotic and the edge of chaos (EOC). We derive exact expressions for the NTK dispersion in the infinite-depth-and-width limit in all three phases and conclude that the NTK variability grows exponentially with depth at the EOC and in the chaotic phase but not in the ordered phase. We also show that the NTK of deep networks may stay constant during training only in the ordered phase and discuss how the structure of the NTK matrix changes during training.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 30 of its 39 references that are in the map. First posted on arXiv 1 Feb 2022.

Builds on

30 of 39 references.

References · 30

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  3. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  4. 2018On Lazy Training in Differentiable Programmingcited by 675
  5. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 634
  6. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  7. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  8. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  9. 2016Deep Information Propagationcited by 195
  10. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  11. 2019Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernelscited by 161
  12. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
18 more
  1. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
  2. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
  3. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
  4. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  5. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  6. 2018Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approachcited by 91
  7. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  8. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
  9. 2019Scaling description of generalization with number of parameters in deep learningcited by 83
  10. 2020The Recurrent Neural Tangent Kernelcited by 67
  11. 2020Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspectivecited by 60
  12. 2019Why bigger is not always better: on finite and infinite neural networkscited by 41
  13. 2020Deep Networks and the Multiple Manifold Problemcited by 35
  14. 2020Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?cited by 33
  15. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  16. 2020Kernel-Based Smoothness Analysis of Residual Networkscited by 19
  17. 2020Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticitycited by 18
  18. 2020Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Widthcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 9; 2024: 2; 2025: 9; 2026: 7.

30 papers.

Sort:

  1. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  2. 2023Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimensioncited by 26
  3. 2024The Challenges of the Nonlinear Regime for Physics-Informed Neural Networkscited by 21
  4. 2023Neural (Tangent Kernel) Collapsecited by 18
  5. 2023How connectivity structure shapes rich and lazy learning in neural circuitscited by 15
  6. 2025Neural Tangent Kernel Analysis to Probe Convergence in Physics-informed Neural Solvers: PIKANs vs. PINNscited by 9
  7. 2022Approximation results for Gradient Descent trained Shallow Neural Networks in 1dcited by 7
  8. 2023Efficient Parametric Approximations of Neural Network Function Space Distancecited by 7
  9. 2026Semantic Tube Prediction: Beating LLM Data Efficiency with JEPAcited by 7
  10. 2025KHRONOS: a Kernel-Based Neural Architecture for Rapid, Resource-Efficient Scientific Computationcited by 5
  11. 2022Engineering flexible machine learning systems by traversing functionally-invariant pathscited by 4
  12. 2023Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networkscited by 4
18 more
  1. 2025Finite-Width Neural Tangent Kernels from Feynman Diagramscited by 4
  2. 2023Approximation Results for Gradient Descent trained Neural Networkscited by 2
  3. 2025GradPCA: Leveraging NTK Alignment for Reliable Out-of-Distribution Detectioncited by 2
  4. 2025Revisiting Glorot Initialization for Long-Range Linear Recurrencescited by 2
  5. 2025Unifying Learning Dynamics and Generalization in Transformers Scaling Lawcited by 2
  6. 2023An Adaptive Tangent Feature Perspective of Neural Networkscited by 1
  7. 2025Depth-induced NTK: Bridging Over-parameterized Neural Networks and Deep Neural Kernelscited by 1
  8. 2025A new initialisation to Control Gradients in Sinusoidal Neural networkcited by 1
  9. 2023The kernel-balanced equation for deep neural networkscited by 0
  10. 2023Infinite Width Graph Neural Networks for Node Regression/ Classificationcited by 0
  11. 2024A simple theory for training response of deep neural networkscited by 0
  12. 2025Scalable Data Attribution via Forward-Only Test-Time Inferencecited by 0
  13. 2026Loop Corrections in Random Feature Models: Training Error and Generalization Gapcited by 0
  14. 2026Man, Machine, and Mathematicscited by 0
  15. 2026When Prompts Interact: Assessing Prompt Arithmetic for Deconfounding under Distribution Shiftcited by 0
  16. 2026How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrencescited by 0
  17. 2026Geometric Dyson Brownian Motions and the Free Log-Normal Limit for a Non-Square Gaussian Matrix Productcited by 0
  18. 2026Kernel Reboot: Breaking the Boundaries of Neural Tangent Kernels for Neural Fieldscited by 0