mlsanityData to 7 Oct 2026

Neural Tangent Kernel: Convergence and Generalization in Neural Networks

arXiv 1806.07572 ↗First posted 20 Jun 2018cs.LG cs.NE math.PRcited by 2,597

Not on the map. Why?

Cited by 35 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit, thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function fθ (which maps input vectors to output vectors) follows the kernel gradient of the functional cost (which is convex, in contrast to the parameter cost) w.r.t. a new kernel: the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANNs. While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and it stays constant during training. This makes it possible to study the training of ANNs in function space instead of parameter space. Convergence of the training can then be related to the positive-definiteness of the limiting NTK. We prove the positive-definiteness of the limiting NTK when the data is supported on the sphere and the non-linearity is non-polynomial. We then focus on the setting of least-squares regression and show that in the infinite-width limit, the network function fθ follows a linear differential equation during training. The convergence is fastest along the largest kernel principal components of the input data with respect to the NTK, hence suggesting a theoretical motivation for early stopping. Finally we study the NTK numerically, observe its behavior for wide networks, and compare it to the infinite-width limit.

Key facts. Cited by 2,597 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 3; 2021: 350; 2022: 402; 2023: 445; 2024: 478; 2025: 478; 2026: 441.

2,597 papers.

Sort:

By area:

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  2. 2022Editing Models with Task Arithmeticcited by 1,076
  3. 2021Robust fine-tuning of zero-shot modelscited by 785
  4. 2021Neural Fields in Visual Computing and Beyondcited by 539
  5. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  6. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  7. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  8. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  9. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  10. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  11. 2021Explaining Neural Scaling Lawscited by 353
  12. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
38 more
  1. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  2. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  3. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  4. 2023Towards Unbounded Machine Unlearningcited by 261
  5. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  6. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  7. 2023WIRE: Wavelet Implicit Neural Representationscited by 220
  8. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  9. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
  10. 2021HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fieldscited by 210
  11. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  12. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  13. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  14. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  15. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  16. 2021Path Integral Sampler: a stochastic control approach for samplingcited by 175
  17. 2021The Values Encoded in Machine Learning Researchcited by 172
  18. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  19. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  20. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  21. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  22. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  23. 2022Respecting causality is all you need for training physics-informed neural networkscited by 154
  24. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  25. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  26. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  27. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  28. 2021Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equationscited by 142
  29. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  30. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  31. 2023Boundary Unlearningcited by 132
  32. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  33. 2021Solving and Learning Nonlinear PDEs with Gaussian Processescited by 128
  34. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  35. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  36. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  37. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  38. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122

50 most cited of 2,597 All 2,597 (JSON)