mlsanityData to 7 Oct 2026

Finite Depth and Width Corrections to the Neural Tangent Kernel

arXiv 1909.05989 ↗First posted 13 Sep 2019cs.LG math.PR stat.MLcited by 114

Not on the map. Why?

Abstract (arXiv)

We prove the precise scaling, at finite depth and width, for the mean and variance of the neural tangent kernel (NTK) in a randomly initialized ReLU network. The standard deviation is exponential in the ratio of network depth to width. Thus, even in the limit of infinite overparameterization, the NTK is not deterministic if depth and width simultaneously tend to infinity. Moreover, we prove that for such deep and wide networks, the NTK has a non-trivial evolution during training by showing that the mean of its first SGD update is also exponential in the ratio of network depth to width. This is sharp contrast to the regime where depth is fixed and network width is very large. Our results suggest that, unlike relatively shallow and wide networks, deep and wide ReLU networks are capable of learning data-dependent features even in the so-called lazy training regime.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 28; 2023: 31; 2024: 10; 2025: 11; 2026: 14.

114 papers.

Sort:

By area:

  1. 2021Explaining Neural Scaling Lawscited by 353
  2. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  3. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  4. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  5. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  6. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  7. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  8. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 57
  9. 2021Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptationcited by 50
  10. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  11. 2023Dataset Distillation with Convexified Implicit Gradientscited by 46
  12. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
38 more
  1. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
  2. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  3. 2023Feature-Learning Networks Are Consistent Across Widths At Realistic Scalescited by 40
  4. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  5. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  6. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  7. 2021Deep Active Learning by Leveraging Training Dynamicscited by 32
  8. 2022Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initializationcited by 30
  9. 2021Offline Neural Contextual Bandits: Pessimism, Optimization and Generalizationcited by 25
  10. 2022The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimescited by 25
  11. 2022Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn featurescited by 24
  12. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  13. 2022On the infinite-depth limit of finite-width neural networkscited by 22
  14. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  15. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  16. 2023Effective Theory of Transformers at Initializationcited by 21
  17. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  18. 2024Towards Stable and Storage-efficient Dataset Distillation: Matching Convexified Trajectorycited by 20
  19. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
  20. 2023Quantitative CLTs in Deep Neural Networkscited by 19
  21. 2023Six Lectures on Linearized Neural Networkscited by 19
  22. 2022Meta-Principled Family of Hyperparameter Scaling Strategiescited by 18
  23. 2023Neural (Tangent Kernel) Collapsecited by 18
  24. 2021Deep ReLU Networks Preserve Expected Lengthcited by 17
  25. 2022Feature Learning in L2-regularized DNNs: Attraction/Repulsion and Sparsitycited by 17
  26. 2025Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfercited by 17
  27. 2021A theory of representation learning gives a deep generalisation of kernel methodscited by 15
  28. 2021Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalizationcited by 15
  29. 2021Convergence of Deep ReLU Networkscited by 14
  30. 2022Evolution of Neural Tangent Kernels under Benign and Adversarial Trainingcited by 14
  31. 2023Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillationcited by 14
  32. 2023Structures of Neural Network Effective Theoriescited by 13
  33. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 13
  34. 2024Asymptotics of Learning with Deep Structured (Random) Featurescited by 13
  35. 2022On Feature Learning in Neural Networks with Global Convergence Guaranteescited by 12
  36. 2022Effects of Data Geometry in Early Deep Learningcited by 12
  37. 2022Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regimecited by 11
  38. 2022The Curious Case of Benign Memorizationcited by 11

50 most cited of 114 All 114 (JSON)