mlsanityData to 7 Oct 2026

Neural Tangent Kernel: A Survey

arXiv 2208.13614 ↗First posted 29 Aug 2022cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

A seminal work [Jacot et al., 2018] demonstrated that training a neural network under specific parameterization is equivalent to performing a particular kernel method as width goes to infinity. This equivalence opened a promising direction for applying the results of the rich literature on kernel methods to neural nets which were much harder to tackle. The present survey covers key results on kernel convergence as width goes to infinity, finite-width corrections, applications, and a discussion of the limitations of the corresponding method.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 44 of its 49 references that are in the map. First posted on arXiv 29 Aug 2022.

Builds on

44 of 49 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  5. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  6. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  7. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  8. 2017Deep Neural Networks as Gaussian Processescited by 617
  9. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  10. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  11. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  12. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
32 more
  1. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  2. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  3. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  4. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  5. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  6. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  7. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 139
  8. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 117
  9. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  10. 2020Kernel methods through the roof: handling billions of points efficientlycited by 88
  11. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  12. 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 79
  13. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  14. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  15. 2017The loss surface of deep and wide neural networkscited by 74
  16. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  17. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  18. 2019Quadratic Suffices for Over-parametrization via Matrix Chernoff Boundcited by 58
  19. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  20. 2020Neural Kernels Without Tangentscited by 56
  21. 2019On Connected Sublevel Sets in Deep Learningcited by 51
  22. 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 51
  23. 2020Tensor Programs III: Neural Matrix Lawscited by 47
  24. 2016Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regionscited by 33
  25. 2020Towards NNGP-guided Neural Architecture Searchcited by 24
  26. 2020Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticitycited by 18
  27. 2021Simple, Fast, and Flexible Framework for Matrix Completion with Infinite Width Neural Networkscited by 17
  28. 2021Neural Tangent Kernel Empowered Federated Learningcited by 15
  29. 2021A Note on Connectivity of Sublevel Sets in Deep Learningcited by 8
  30. 2020Dynamically Stable Infinite-Width Limits of Neural Classifierscited by 6
  31. 2021DNN-Based Topology Optimisation: Spatial Invariance and Neural Tangent Kernelcited by 2
  32. 2020Notes on Deep Learning Theorycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 5; 2025: 4; 2026: 6.

22 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  2. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 57
  3. 2023Fast Graph Condensation with Structure-based Neural Tangent Kernelcited by 36
  4. 2024Emergent Equivariance in Deep Ensemblescited by 18
  5. 2023Regularize implicit neural representation by itselfcited by 13
  6. 2023How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient Descentcited by 10
  7. 2024Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyondcited by 9
  8. 2024Optimal Convergence Rates for Neural Operatorscited by 7
  9. 2025A Unified Theory of Sinusoidal Activation Families for Implicit Neural Representationscited by 4
  10. 2023How Graph Neural Networks Learn: Lessons from Training Dynamicscited by 3
  11. 2024Epistemic Uncertainty and Observation Noise with the Neural Tangent Kernelcited by 3
  12. 2025AdaPM: a Partial Momentum Algorithm for LLM Trainingcited by 3
10 more
  1. 2022Statistical Physics of Deep Neural Networks: Initialization toward Optimal Channelscited by 2
  2. 2025Just One Layer Norm Guarantees Stable Extrapolationcited by 2
  3. 2026Efficient Adversarial Attacks on High-dimensional Offline Banditscited by 2
  4. 2026Delving into Spectral Clustering with Vision-Language Representationscited by 2
  5. 2024Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theoriescited by 0
  6. 2025Random at First, Fast at Last: NTK-Guided Fourier Pre-Processing for Tabular DLcited by 0
  7. 2026CogGen: Cognitive-Load-Inspired Fully Unsupervised Deep Generative Modeling for Compressively Sampled MRI Reconstructioncited by 0
  8. 2026RAMPAGE: RAndomized Mid-Point for debiAsed Gradient Extrapolationcited by 0
  9. 2026Data-driven Koopman mode approximation: A neural power iteration algorithmcited by 0
  10. 2026Relative Generalization Invariance of LLM Pretrainingcited by 0