mlsanityData to 14 Oct 2026

Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel

arXiv 2010.15110 ↗First posted 28 Oct 2020cs.LG stat.MLcited by 187

Not on the map. Why?

Abstract (arXiv)

In suitably initialized wide networks, small learning rates transform deep neural networks (DNNs) into neural tangent kernel (NTK) machines, whose training dynamics is well-approximated by a linear weight expansion of the network at initialization. Standard training, however, diverges from its linearization in ways that are poorly understood. We study the relationship between the training dynamics of nonlinear deep networks, the geometry of the loss landscape, and the time evolution of a data-dependent NTK. We do so through a large-scale phenomenological analysis of training, synthesizing diverse measures characterizing loss landscape geometry and NTK dynamics. In multiple neural architectures and datasets, we find these diverse measures evolve in a highly correlated manner, revealing a universal picture of the deep learning process. In this picture, deep network training exhibits a highly chaotic rapid initial transient that within 2 to 3 epochs determines the final linearly connected basin of low loss containing the end point of training. During this chaotic transient, the NTK changes rapidly, learning useful features from the training data that enables it to outperform the standard initial NTK by a factor of 3 in less than 3 to 4 epochs. After this rapid chaotic transient, the NTK changes at constant velocity, and its performance matches that of full network training in 15% to 45% of training time. Overall, our analysis reveals a striking correlation between a diverse set of metrics over training time, governed by a rapid chaotic to stable transition in the first few epochs, that together poses challenges and opportunities for the development of more accurate theories of deep learning.

Key facts. Cited by 187 papers in the map (data through 14 Oct 2026). First posted on arXiv 28 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 24; 2022: 45; 2023: 31; 2024: 35; 2025: 32; 2026: 20.

187 papers.

Sort:

By area:

  1. 2022Editing Models with Task Arithmeticcited by 1,120
  2. 2021Robust fine-tuning of zero-shot modelscited by 797
  3. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  4. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 221
  5. 2022Patching open-vocabulary models by interpolating weightscited by 190
  6. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  7. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 127
  8. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 112
  9. 2024A Dynamical Model of Neural Scaling Lawscited by 111
  10. 2023A Spectral Condition for Feature Learningcited by 109
  11. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  12. 2022Dataset Condensation with Contrastive Signalscited by 105
38 more
  1. 2024Learning Dynamics of LLM Finetuningcited by 95
  2. 2023Deep Model Fusion: A Surveycited by 91
  3. 2021Improved architectures and training algorithms for deep operator networkscited by 86
  4. 2021Learning Neural Network Subspacescited by 82
  5. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  6. 2024Normalization and effective learning rates in reinforcement learningcited by 58
  7. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 56
  8. 2023Towards Last-layer Retraining for Group Robustness with Fewer Annotationscited by 54
  9. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  10. 2023Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivitycited by 50
  11. 2024Scaling and renormalization in high-dimensional regressioncited by 50
  12. 2021What can linearized neural networks actually say about generalization?cited by 49
  13. 2022A Framework and Benchmark for Deep Batch Active Learning for Regressioncited by 49
  14. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 49
  15. 2023Dataset Distillation with Convexified Implicit Gradientscited by 46
  16. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 42
  17. 2023Feature-Learning Networks Are Consistent Across Widths At Realistic Scalescited by 41
  18. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 41
  19. 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 38
  20. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  21. 2021On the Convergence and Calibration of Deep Learning with Differential Privacycited by 36
  22. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  23. 2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 36
  24. 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 35
  25. 2022Limitations of the NTK for Understanding Generalization in Deep Learningcited by 35
  26. 2023Data Valuation Without Training of a Modelcited by 35
  27. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
  28. 2021Towards Understanding the Condensation of Neural Networks at Initial Trainingcited by 32
  29. 2021Deep Active Learning by Leveraging Training Dynamicscited by 32
  30. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 31
  31. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 31
  32. 2022Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initializationcited by 30
  33. 2022A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernelcited by 30
  34. 2022A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biasescited by 30
  35. 2022Not All Poisons are Created Equal: Robust Training against Data Poisoningcited by 30
  36. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  37. 2021Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapescited by 28
  38. 2021How many degrees of freedom do we need to train deep networks: a loss landscape perspectivecited by 28

50 most cited of 187 All 187 (JSON)