Reactivation: Empirical NTK Dynamics Under Task Shifts
Not on the map. Why?
Abstract (arXiv)
The Neural Tangent Kernel (NTK) offers a powerful tool to study the functional dynamics of neural networks. In the so-called lazy, or kernel regime, the NTK remains static during training and the network function is linear in the static neural tangents feature space. The evolution of the NTK during training is necessary for feature learning, a key driver of deep learning success. The study of the NTK dynamics has led to several critical discoveries in recent years, in generalization and scaling behaviours. However, this body of work has been limited to the single task setting, where the data distribution is assumed constant over time. In this work, we present a comprehensive empirical analysis of NTK dynamics in continual learning, where the data distribution shifts over time. Our findings highlight continual learning as a rich and underutilized testbed for probing the dynamics of neural training. At the same time, they challenge the validity of static-kernel approximations in theoretical treatments of continual learning, even at large scale.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 13 of its 17 references that are in the map. First posted on arXiv 21 Jul 2025.
Builds on
13 of 17 references.
References · 13
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
- 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
- 2020Feature Learning in Infinite-Width Neural Networkscited by 182
- 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
- 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 139
- 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 99
- 2020Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descentcited by 68
- 2021A Theory of Neural Tangent Kernel Alignment and Its Influence on Trainingcited by 25
- 2021Learning Curves for Continual Learning in Neural Networks: Self-Knowledge Transfer and Forgettingcited by 14
1 more
- 2025On the Cone Effect in the Learning Dynamicscited by 2
Built on by
Not cited yet.