mlsanityData to 14 Oct 2026

Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stability

arXiv 2507.12837 ↗First posted 17 Jul 2025cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

The study of Neural Tangent Kernels (NTKs) in deep learning has drawn increasing attention in recent years. NTKs typically actively change during training and are related to feature learning. In parallel, recent work on Gradient Descent (GD) has found a phenomenon called Edge of Stability (EoS), in which the largest eigenvalue of the NTK oscillates around a value inversely proportional to the step size. However, although follow-up works have explored the underlying mechanism of such eigenvalue behavior in depth, the understanding of the behavior of the NTK eigenvectors during EoS is still missing. This paper examines the dynamics of NTK eigenvectors during EoS in detail. Across different architectures, we observe that larger learning rates cause the leading eigenvectors of the final NTK, as well as the full NTK matrix, to have greater alignment with the training target. We then study the underlying mechanism of this phenomenon and provide a theoretical analysis for a two-layer linear network. Our study enhances the understanding of GD training dynamics in deep learning.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 25 of its 37 references that are in the map. First posted on arXiv 17 Jul 2025.

Builds on

25 of 37 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  5. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  6. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  7. 2017Implicit Regularization in Matrix Factorizationcited by 322
  8. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  9. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  10. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  11. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  12. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
13 more
  1. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  2. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  3. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  4. 2021Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristicscited by 67
  5. 2024Understanding Optimization in Deep Learning with Central Flowscited by 64
  6. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 63
  7. 2021What can linearized neural networks actually say about generalization?cited by 49
  8. 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
  9. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 31
  10. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  11. 2021A Theory of Neural Tangent Kernel Alignment and Its Influence on Trainingcited by 25
  12. 2022Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn featurescited by 25
  13. 2022How Does Adaptive Optimization Impact Local Neural Network Geometry?cited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026A Defense of the Quadratic Modelcited by 14
  2. 2026Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stabilitycited by 9
  3. 2025Mathematical Foundations of Neural Tangents and Infinite-Width Networkscited by 1
  4. 2026Does Weight Decay Enhance Training Stability?cited by 1
  5. 2026The Road Taken: The Role of Optimizers at the Edge of Stabilitycited by 1
  6. 2026Label-NTK Alignments and A Tighter Convergence Bound in the NTK Regimecited by 0