mlsanityData to 14 Oct 2026

On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspective

arXiv 2310.00137 ↗First posted 29 Sep 2023cs.LG stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

The neural tangent kernel (NTK) has garnered significant attention as a theoretical framework for describing the behavior of large-scale neural networks. Kernel methods are theoretically well-understood and as a result enjoy algorithmic benefits, which can be demonstrated to hold in wide synthetic neural network architectures. These advantages include faster optimization, reliable uncertainty quantification and improved continual learning. However, current results quantifying the rate of convergence to the kernel regime suggest that exploiting these benefits requires architectures that are orders of magnitude wider than they are deep. This assumption raises concerns that architectures used in practice do not exhibit behaviors as predicted by the NTK. Here, we supplement previous work on the NTK by empirically investigating whether the limiting regime predicts practically relevant behavior of large-width architectures. Our results demonstrate that this is not the case across multiple domains. This observed disconnect between theory and practice further calls into question to what degree NTK theory should inform architectural and algorithmic choices.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 51 of its 67 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

51 of 67 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  6. 2016Wide Residual Networkscited by 3,070
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  8. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  9. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  10. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  11. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  12. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
38 more
  1. 2017Deep Neural Networks as Gaussian Processescited by 622
  2. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  3. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  4. 2019Orthogonal Gradient Descent for Continual Learningcited by 369
  5. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  6. 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
  7. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  8. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  9. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
  10. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  11. 2020Neural Thompson Samplingcited by 128
  12. 2021Avalanche: an End-to-End Library for Continual Learningcited by 124
  13. 2019Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentencecited by 122
  14. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 119
  15. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 106
  16. 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 103
  17. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  18. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  19. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 81
  20. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  21. 2020Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descentcited by 68
  22. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  23. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  24. 2023A Study of Bayesian Neural Network Surrogates for Bayesian Optimizationcited by 50
  25. 2021Neural Contextual Bandits without Regretcited by 40
  26. 2020Deep Networks and the Multiple Manifold Problemcited by 35
  27. 2023Continual Learning in Linear Classification on Separable Datacited by 34
  28. 2022Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regimecited by 29
  29. 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
  30. 2021Offline Neural Contextual Bandits: Pessimism, Optimization and Generalizationcited by 25
  31. 2022Wide Mean-Field Bayesian Neural Networks Ignore the Datacited by 22
  32. 2019Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theorycited by 21
  33. 2021The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspectivecited by 21
  34. 2023The Geometry of Neural Nets' Parameter Spaces Under Reparametrizationcited by 16
  35. 2021Being a Bit Frequentist Improves Bayesian Neural Networkscited by 15
  36. 2022Posterior Refinement Improves Sample Efficiency in Bayesian Neural Networkscited by 14
  37. 2022Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regimecited by 11
  38. 2023Promises and Pitfalls of the Linearized Laplace in Bayesian Optimizationcited by 10

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 5.

6 papers.

  1. 2024The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Modelcited by 26
  2. 2025Finite-Width Neural Tangent Kernels from Feynman Diagramscited by 4
  3. 2025Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networkscited by 3
  4. 2025On the Theory of Continual Learning with Gradient Descent for Neural Networkscited by 3
  5. 2025Changing Base Without Losing Pace: A GPU-Efficient Alternative to MatMul in DNNscited by 2
  6. 2025Feature learning is decoupled from generalization in high capacity neural networkscited by 1