mlsanityData to 7 Oct 2026

Tight conditions for when the NTK approximation is valid

arXiv 2305.13141 ↗First posted 22 May 2023cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

We study when the neural tangent kernel (NTK) approximation is valid for training a model with the square loss. In the lazy training setting of Chizat et al. 2019, we show that rescaling the model by a factor of α = O(T) suffices for the NTK approximation to be valid until training time T. Our bound is tight and improves on the previous bound of Chizat et al. 2019, which required a larger rescaling factor of α = O(T2).

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 38 of its 46 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

38 of 46 references.

References · 38

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  3. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  7. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  8. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  9. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  10. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  11. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  12. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
26 more
  1. 2020On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networkscited by 279
  2. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  3. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  4. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  5. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
  6. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  7. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  8. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  9. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  10. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  11. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
  12. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  13. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  14. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  15. 2021Learning with invariances in random features and kernel modelscited by 89
  16. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  17. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  18. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  19. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  20. 2019Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networkscited by 60
  21. 2020Towards Understanding Hierarchical Learning: Benefits of Neural Representationscited by 46
  22. 2019Mean Field Analysis of Deep Neural Networkscited by 42
  23. 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
  24. 2022Learning sparse features can lead to overfitting in neural networkscited by 23
  25. 2022Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomialscited by 13
  26. 2020Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Widthcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The role of parameter Jacobians in the stability of network outputscited by 0