mlsanityData to 7 Oct 2026

On the linearity of large non-linear models: when and why the tangent kernel is constant

arXiv 2010.01092 ↗First posted 2 Oct 2020cs.LG stat.MLcited by 129

Not on the map. Why?

Abstract (arXiv)

The goal of this work is to shed light on the remarkable phenomenon of transition to linearity of certain neural networks as their width approaches infinity. We show that the transition to linearity of the model and, equivalently, constancy of the (neural) tangent kernel (NTK) result from the scaling properties of the norm of the Hessian matrix of the network as a function of the network width. We present a general framework for understanding the constancy of the tangent kernel via Hessian scaling applicable to the standard classes of neural networks. Our analysis provides a new perspective on the phenomenon of constant tangent kernel, which is different from the widely accepted "lazy training". Furthermore, we show that the transition to linearity is not a general property of wide neural networks and does not hold when the last layer of the network is non-linear. It is also not necessary for successful optimization by gradient descent.

Key facts. Cited by 129 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 30; 2023: 22; 2024: 29; 2025: 15; 2026: 13.

129 papers.

Sort:

By area:

  1. 2022How Do Vision Transformers Work?cited by 305
  2. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  3. 2024Challenges in Training PINNs: A Loss Landscape Perspectivecited by 109
  4. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  5. 2024Simplifying Deep Temporal Difference Learningcited by 71
  6. 2024Scalable Optimization in the Modular Normcited by 64
  7. 2021Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Timecited by 62
  8. 2021Score-based Generative Neural Networks for Large-Scale Optimal Transportcited by 59
  9. 2021Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depthcited by 55
  10. 2021What can linearized neural networks actually say about generalization?cited by 49
  11. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  12. 2021Early Stopping for Deep Image Priorcited by 44
38 more
  1. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
  2. 2022Graph Neural Networks are Inherently Good Generalizers: Insights by Bridging GNNs and MLPscited by 42
  3. 2024Numerical analysis of physics-informed neural networks and related models in physics-informed machine learningcited by 39
  4. 2023On the Optimization and Generalization of Multi-head Attentioncited by 37
  5. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
  6. 2021Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapescited by 27
  7. 2022Do Residual Neural Networks discretize Neural Ordinary Differential Equations?cited by 27
  8. 2023An operator preconditioning perspective on training in physics-informed machine learningcited by 27
  9. 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
  10. 2021Stability & Generalisation of Gradient Descent for Shallow Neural Networks without the Neural Tangent Kernelcited by 25
  11. 2022On the Generalization Mystery in Deep Learningcited by 25
  12. 2021Overparameterization of deep ResNet: zero loss and mean-field analysiscited by 24
  13. 2022Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn featurescited by 24
  14. 2023Aiming towards the minimizers: fast convergence of SGD for overparametrized problemscited by 22
  15. 2021A Neural Tangent Kernel Perspective of GANscited by 21
  16. 2021Uniform Generalization Bounds for Overparameterized Neural Networkscited by 21
  17. 2022What Can the Neural Tangent Kernel Tell Us About Adversarial Robustness?cited by 21
  18. 2023Unifying GANs and Score-Based Diffusion as Generative Particle Modelscited by 21
  19. 2024The Challenges of the Nonlinear Regime for Physics-Informed Neural Networkscited by 21
  20. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  21. 2023Six Lectures on Linearized Neural Networkscited by 19
  22. 2022Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?cited by 18
  23. 2021On the Equivalence between Neural Network and Support Vector Machinecited by 17
  24. 2022Quadratic models for understanding catapult dynamics of neural networkscited by 17
  25. 2024Reparameterization invariance in approximate Bayesian inferencecited by 17
  26. 2021Learning with Gradient Descent and Weakly Convex Lossescited by 16
  27. 2023Graph Neural Tangent Kernel: Convergence on Large Graphscited by 16
  28. 2023On the Convergence of Encoder-only Shallow Transformerscited by 13
  29. 2024Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimensioncited by 13
  30. 2022Wide and Deep Neural Networks Achieve Optimality for Classificationcited by 12
  31. 2022Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regimecited by 11
  32. 2023Non-Parametric Representation Learning with Kernelscited by 11
  33. 2021On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regimecited by 10
  34. 2022Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networkscited by 9
  35. 2024Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyondcited by 9
  36. 2022Kernel Methods and Multi-layer Perceptrons Learn Linear Models in High Dimensionscited by 8
  37. 2024Peer-to-Peer Learning Dynamics of Wide Neural Networkscited by 8
  38. 2025Enhancing Physics-Informed Neural Networks Through Feature Engineeringcited by 8

50 most cited of 129 All 129 (JSON)