mlsanityData to 7 Oct 2026

Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent

arXiv 1902.06720 ↗First posted 18 Feb 2019stat.ML cs.LGcited by 732

Not on the map. Why?

Abstract (arXiv)

A longstanding goal in deep learning research has been to precisely characterize training and generalization. However, the often complex loss landscapes of neural networks have made a theory of learning dynamics elusive. In this work, we show that for wide neural networks the learning dynamics simplify considerably and that, in the infinite width limit, they are governed by a linear model obtained from the first-order Taylor expansion of the network around its initial parameters. Furthermore, mirroring the correspondence between wide Bayesian neural networks and Gaussian processes, gradient-based training of wide neural networks with a squared loss produces test set predictions drawn from a Gaussian process with a particular compositional kernel. While these theoretical results are only exact in the infinite width limit, we nevertheless find excellent empirical agreement between the predictions of the original network and those of the linearized version even for finite practically-sized networks. This agreement is robust across different architectures, optimization methods, and loss functions.

Key facts. Cited by 732 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 114; 2022: 148; 2023: 134; 2024: 120; 2025: 114; 2026: 101.

732 papers.

Sort:

By area:

  1. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  2. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  3. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  4. 2021Explaining Neural Scaling Lawscited by 353
  5. 2023WIRE: Wavelet Implicit Neural Representationscited by 220
  6. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  7. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
  8. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  9. 2021The Values Encoded in Machine Learning Researchcited by 172
  10. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  11. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  12. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
38 more
  1. 2022If Influence Functions are the Answer, Then What is the Question?cited by 142
  2. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  3. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  4. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  5. 2023An Expert's Guide to Training Physics-informed Neural Networkscited by 119
  6. 2023Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matchingcited by 118
  7. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  8. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  9. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  10. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  11. 2022A Solvable Model of Neural Scaling Lawscited by 101
  12. 2023A Spectral Condition for Feature Learningcited by 101
  13. 2024Learning Dynamics of LLM Finetuningcited by 93
  14. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  15. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
  16. 2021Improved architectures and training algorithms for deep operator networkscited by 85
  17. 2022Understanding the unstable convergence of gradient descentcited by 84
  18. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  19. 2023DataDAM: Efficient Dataset Distillation with Attention Matchingcited by 83
  20. 2023Grokking modular arithmeticcited by 77
  21. 2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 75
  22. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
  23. 2022Universality of empirical risk minimizationcited by 73
  24. 2021On the geometry of generalization and memorization in deep neural networkscited by 71
  25. 2024Simplifying Deep Temporal Difference Learningcited by 71
  26. 2024Modular Duality in Deep Learningcited by 71
  27. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  28. 2022Overview frequency principle/spectral bias in deep learningcited by 68
  29. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  30. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  31. 2021Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Timecited by 62
  32. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  33. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
  34. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  35. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 57
  36. 2021Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depthcited by 55
  37. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  38. 2024MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmeticcited by 53

50 most cited of 732 All 732 (JSON)