On the Cone Effect in the Learning Dynamics
Not on the map. Why?
Abstract (arXiv)
Understanding the learning dynamics of neural networks is a central topic in the deep learning community. In this paper, we take an empirical perspective to study the learning dynamics of neural networks in real-world settings. Specifically, we investigate the evolution process of the empirical Neural Tangent Kernel (eNTK) during training. Our key findings reveal a two-phase learning process: i) in Phase I, the eNTK evolves significantly, signaling the rich regime, and ii) in Phase II, the eNTK keeps evolving but is constrained in a narrow space, a phenomenon we term the cone effect. This two-phase framework builds on the hypothesis proposed by Fort et al. (2020), but we uniquely identify the cone effect in Phase II, demonstrating its significant performance advantages over fully linearized training.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 25 of its 33 references that are in the map. First posted on arXiv 20 Mar 2025.
Builds on
25 of 33 references.
References · 25
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
- 2018On Lazy Training in Differentiable Programmingcited by 675
- 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
- 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
- 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
- 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
- 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
13 more
- 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
- 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
- 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
- 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
- 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
- 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85
- 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 70
- 2023Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivitycited by 50
- 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 49
- 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 42
- 2020On the training dynamics of deep networks with L2 regularizationcited by 39
- 2022Feature Learning in L2-regularized DNNs: Attraction/Repulsion and Sparsitycited by 17
- 2023How connectivity structure shapes rich and lazy learning in neural circuitscited by 15
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.