mlsanityData to 14 Oct 2026

Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?

arXiv 2012.04477 ↗First posted 8 Dec 2020cs.LG stat.MLcited by 33

Not on the map. Why?

Abstract (arXiv)

Neural Tangent Kernel (NTK) theory is widely used to study the dynamics of infinitely-wide deep neural networks (DNNs) under gradient descent. But do the results for infinitely-wide networks give us hints about the behavior of real finite-width ones? In this paper, we study empirically when NTK theory is valid in practice for fully-connected ReLU and sigmoid DNNs. We find out that whether a network is in the NTK regime depends on the hyperparameters of random initialization and the network's depth. In particular, NTK theory does not explain the behavior of sufficiently deep networks initialized so that their gradients explode as they propagate through the network's layers: the kernel is random at initialization and changes significantly during training in this case, contrary to NTK theory. On the other hand, in the case of vanishing gradients, DNNs are in the the NTK regime but become untrainable rapidly with depth. We also describe a framework to study generalization properties of DNNs, in particular the variance of network's output function, by means of NTK theory and discuss its limits.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). First posted on arXiv 8 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 9; 2023: 6; 2024: 5; 2025: 7; 2026: 4.

33 papers.

Sort:

  1. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  2. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  3. 2022Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initializationcited by 30
  4. 2021Stability & Generalisation of Gradient Descent for Shallow Neural Networks without the Neural Tangent Kernelcited by 26
  5. 2022On the infinite-depth limit of finite-width neural networkscited by 22
  6. 2022Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networkscited by 21
  7. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  8. 2024The Challenges of the Nonlinear Regime for Physics-Informed Neural Networkscited by 21
  9. 2023Neural (Tangent Kernel) Collapsecited by 18
  10. 2022Bounding the Rademacher Complexity of Fourier neural operatorscited by 16
  11. 2022Few-shot Backdoor Attacks via Neural Tangent Kernelscited by 15
  12. 2022Joint Embedding Self-Supervised Learning in the Kernel Regimecited by 12
21 more
  1. 2022Gaussian Pre-Activations in Neural Networks: Myth or Reality?cited by 9
  2. 2025Neural Tangent Kernel Analysis to Probe Convergence in Physics-informed Neural Solvers: PIKANs vs. PINNscited by 9
  3. 2023Efficient Parametric Approximations of Neural Network Function Space Distancecited by 7
  4. 2024Improving Adaptivity via Over-Parameterization in Sequence Modelscited by 3
  5. 2025Beyond Scaling Curves: Internal Dynamics of Neural Networks Through the NTK Lenscited by 2
  6. 2022On Connecting Deep Trigonometric Networks with Deep Gaussian Processes: Covariance, Expressivity, and Neural Tangent Kernelcited by 1
  7. 2023An Adaptive Tangent Feature Perspective of Neural Networkscited by 1
  8. 2023Commutative Width and Depth Scaling in Deep Neural Networkscited by 1
  9. 2024Probability Distribution Learning and Its Application in Deep Learningcited by 1
  10. 2025Issues with Neural Tangent Kernel Approach to Neural Networkscited by 1
  11. 2025Feature learning is decoupled from generalization in high capacity neural networkscited by 1
  12. 2023The kernel-balanced equation for deep neural networkscited by 0
  13. 2024GradAlign for Training-free Model Performance Inferencecited by 0
  14. 2024Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theoriescited by 0
  15. 2025Towards Understanding the Optimization Mechanisms in Deep Learningcited by 0
  16. 2025Compact Vision Transformer by Reduction of Kernel Complexitycited by 0
  17. 2025Revisiting the Neural Tangent Kernel: the role of large width and depthcited by 0
  18. 2026On the Equivalence of Random Network Distillation, Deep Ensembles, and Bayesian Inferencecited by 0
  19. 2026Man, Machine, and Mathematicscited by 0
  20. 2026Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothnesscited by 0
  21. 2026The role of parameter Jacobians in the stability of network outputscited by 0