mlsanityData to 7 Oct 2026

Limitations of the NTK for Understanding Generalization in Deep Learning

arXiv 2206.10012 ↗First posted 20 Jun 2022cs.LG cs.AIcited by 34

Not on the map. Why?

Abstract (arXiv)

The ``Neural Tangent Kernel'' (NTK) (Jacot et al 2018), and its empirical variants have been proposed as a proxy to capture certain behaviors of real neural networks. In this work, we study NTKs through the lens of scaling laws, and demonstrate that they fall short of explaining important aspects of neural network generalization. In particular, we demonstrate realistic settings where finite-width neural networks have significantly better data scaling exponents as compared to their corresponding empirical and infinite NTKs at initialization. This reveals a more fundamental difference between the real networks and NTKs, beyond just a few percentage points of test accuracy. Further, we show that even if the empirical NTK is allowed to be pre-trained on a constant number of samples, the kernel scaling does not catch up to the neural network scaling. Finally, we show that the empirical NTK continues to evolve throughout most of the training, in contrast with prior work which suggests that it stabilizes after a few epochs of training. Altogether, our work establishes concrete limitations of the NTK approach in understanding generalization of real networks on natural datasets.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 36 of its 48 references that are in the map. First posted on arXiv 20 Jun 2022.

Builds on

36 of 48 references.

References · 36

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  5. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  6. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  7. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  8. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  9. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  10. 2021Explaining Neural Scaling Lawscited by 353
  11. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  12. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
24 more
  1. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  2. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  3. 2021The Principles of Deep Learning Theorycited by 169
  4. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  5. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  6. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  7. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  8. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  9. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  10. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  11. 2020Learning Parities with Neural Networkscited by 77
  12. 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
  13. 2020Triple descent and the two kinds of overfitting: Where & why do they appear?cited by 64
  14. 2018The jamming transition as a paradigm to understand the loss landscape of deep neural networkscited by 61
  15. 2020Neural Kernels Without Tangentscited by 56
  16. 2022Data Scaling Laws in NMT: The Effect of Noise and Architecturecited by 55
  17. 2021What can linearized neural networks actually say about generalization?cited by 49
  18. 2020A Neural Scaling Law from the Dimension of the Data Manifoldcited by 48
  19. 2020Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networkscited by 44
  20. 2019Mean Field Analysis of Deep Neural Networkscited by 42
  21. 2021Towards an Understanding of Benign Overfitting in Neural Networkscited by 29
  22. 2020Geometric compression of invariant manifolds in neural netscited by 28
  23. 2021Properties of the After Kernelcited by 27
  24. 2020Asymptotics of Wide Convolutional Neural Networkscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 11; 2024: 6; 2025: 7; 2026: 5.

34 papers.

Sort:

  1. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  2. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  3. 2023A Spectral Condition for Feature Learningcited by 101
  4. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  5. 2023Feature-Learning Networks Are Consistent Across Widths At Realistic Scalescited by 40
  6. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  7. 2023Generalization Ability of Wide Neural Networks on ℝcited by 28
  8. 2022The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimescited by 25
  9. 2022Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn featurescited by 24
  10. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  11. 2023The Law of Parsimony in Gradient Descent for Learning Deep Linear Networkscited by 19
  12. 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
22 more
  1. 2024When does compositional structure yield compositional generalization? A kernel theorycited by 19
  2. 2023Mechanism of feature learning in convolutional neural networkscited by 15
  3. 2024The Optimization Landscape of SGD Across the Feature Learning Strengthcited by 13
  4. 2025Adaptive kernel predictors from feature-learning infinite limits of neural networkscited by 13
  5. 2023Faithful and Efficient Explanations for Neural Networks via Neural Tangent Kernel Surrogate Modelscited by 8
  6. 2023Connecting NTK and NNGP: A Unified Theoretical Framework for Wide Neural Network Learning Dynamicscited by 6
  7. 2026A Defense of the Quadratic Modelcited by 6
  8. 2024Task structure and nonlinearity jointly determine learned representational geometrycited by 5
  9. 2026A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuningcited by 5
  10. 2022Behind the Scenes of Gradient Descent: A Trajectory Analysis via Basis Function Decompositioncited by 4
  11. 2025Divergence of Empirical Neural Tangent Kernel in Classification Problemscited by 3
  12. 2025Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networkscited by 3
  13. 2023Approximation Results for Gradient Descent trained Neural Networkscited by 2
  14. 2024Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspectivecited by 2
  15. 2022On Kernel Regression with Data-Dependent Kernelscited by 1
  16. 2025Feature learning is decoupled from generalization in high capacity neural networkscited by 1
  17. 2025Harnessing Optimization Dynamics for Curvature-Informed Model Mergingcited by 1
  18. 2022Understanding Sparse Feature Updates in Deep Networks using Iterative Linearisationcited by 0
  19. 2025Towards Understanding the Optimization Mechanisms in Deep Learningcited by 0
  20. 2025Training NTK to Generalize with KAREcited by 0
  21. 2026The role of parameter Jacobians in the stability of network outputscited by 0
  22. 2026Kernel Reboot: Breaking the Boundaries of Neural Tangent Kernels for Neural Fieldscited by 0