mlsanityData to 7 Oct 2026

Tensor Programs II: Neural Tangent Kernel for Any Architecture

arXiv 2006.14548 ↗First posted 25 Jun 2020stat.ML cond-mat.dis-nn cs.LGcited by 139

Not on the map. Why?

Abstract (arXiv)

We prove that a randomly initialized neural network of *any architecture* has its Tangent Kernel (NTK) converge to a deterministic limit, as the network widths tend to infinity. We demonstrate how to calculate this limit. In prior literature, the heuristic study of neural network gradients often assumes every weight matrix used in forward propagation is independent from its transpose used in backpropagation (Schoenholz et al. 2017). This is known as the *gradient independence assumption (GIA)*. We identify a commonly satisfied condition, which we call *Simple GIA Check*, such that the NTK limit calculation based on GIA is correct. Conversely, when Simple GIA Check fails, we show GIA can result in wrong answers. Our material here presents the NTK results of Yang (2019a) in a friendly manner and showcases the *tensor programs* technique for understanding wide neural networks. We provide reference implementations of infinite-width NTKs of recurrent neural network, transformer, and batch normalization at https://github.com/thegregyang/NTK4A.

Key facts. Cited by 139 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 24; 2023: 31; 2024: 20; 2025: 27; 2026: 20.

139 papers.

Sort:

By area:

  1. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  2. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  3. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
  4. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  5. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  6. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  7. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  8. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  9. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  10. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  11. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  12. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
38 more
  1. 2023How to Capture Higher-order Correlations? Generalizing Matrix Softmax Attention to Kronecker Computationcited by 44
  2. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
  3. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  4. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  5. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
  6. 2023Deep networks for system identification: a Surveycited by 35
  7. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  8. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimationcited by 33
  9. 2022Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initializationcited by 30
  10. 2022A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernelcited by 30
  11. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  12. 2022Making Look-Ahead Active Learning Strategies Feasible with Neural Tangent Kernelscited by 25
  13. 2023Neural signature kernels as infinite-width-depth-limits of controlled ResNetscited by 25
  14. 2021Nonperturbative renormalization for the neural network-QFT correspondencecited by 24
  15. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  16. 2022Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regressioncited by 22
  17. 2022Neural Tangent Kernel: A Surveycited by 22
  18. 2021A Neural Tangent Kernel Perspective of GANscited by 21
  19. 2021The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspectivecited by 21
  20. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  21. 2023Neural Network Field Theories: Non-Gaussianity, Actions, and Localitycited by 21
  22. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  23. 2023Quantitative CLTs in Deep Neural Networkscited by 19
  24. 2022Overcoming the Spectral Bias of Neural Value Approximationcited by 18
  25. 2023Neural (Tangent Kernel) Collapsecited by 18
  26. 2024Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimescited by 18
  27. 2022Empirical Phase Diagram for Three-layer Neural Networks with Infinite Widthcited by 17
  28. 2021Shift Invariance Can Reduce Adversarial Robustnesscited by 16
  29. 2022Auto-scaling Vision Transformers without Trainingcited by 15
  30. 2022Few-shot Backdoor Attacks via Neural Tangent Kernelscited by 15
  31. 2022An Analysis of Attention via the Lens of Exchangeability and Latent Variable Modelscited by 15
  32. 2023How connectivity structure shapes rich and lazy learning in neural circuitscited by 15
  33. 2021Towards Deepening Graph Neural Networks: A GNTK-based Optimization Perspectivecited by 14
  34. 2023On the Parameterization of Second-Order Optimization Effective Towards the Infinite Widthcited by 14
  35. 2023Regularize implicit neural representation by itselfcited by 13
  36. 2023Structures of Neural Network Effective Theoriescited by 13
  37. 2023On the Convergence of Encoder-only Shallow Transformerscited by 13
  38. 2022A Kernel Perspective of Skip Connections in Convolutional Networkscited by 12

50 most cited of 139 All 139 (JSON)