mlsanityData to 14 Oct 2026

New Evidence of the Two-Phase Learning Dynamics of Neural Networks

arXiv 2505.13900 ↗First posted 20 May 2025cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Understanding how deep neural networks learn remains a fundamental challenge in modern machine learning. A growing body of evidence suggests that training dynamics undergo a distinct phase transition, yet our understanding of this transition is still incomplete. In this paper, we introduce an interval-wise perspective that compares network states across a time window, revealing two new phenomena that illuminate the two-phase nature of deep learning. i) The Chaos Effect. By injecting an imperceptibly small parameter perturbation at various stages, we show that the response of the network to the perturbation exhibits a transition from chaotic to stable, suggesting there is an early critical period where the network is highly sensitive to initial conditions; ii) The Cone Effect. Tracking the evolution of the empirical Neural Tangent Kernel (eNTK), we find that after this transition point the model's functional trajectory is confined to a narrow cone-shaped subset: while the kernel continues to change, it gets trapped into a tight angular region. Together, these effects provide a structural, dynamical view of how deep networks transition from sensitive exploration to stable refinement during training.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 32 of its 37 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

32 of 37 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  5. 2021Robust fine-tuning of zero-shot modelscited by 797
  6. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  7. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  8. 2018On Lazy Training in Differentiable Programmingcited by 675
  9. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  10. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  11. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  12. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
20 more
  1. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  2. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  3. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  4. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  5. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 221
  6. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  7. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  8. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 176
  9. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  10. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  11. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
  12. 2021Assessing Generalization of SGD via Disagreementcited by 117
  13. 2020The Early Phase of Neural Network Trainingcited by 115
  14. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  15. 2017Critical Learning Periods in Deep Neural Networkscited by 79
  16. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 63
  17. 2023Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivitycited by 50
  18. 2024On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigmcited by 26
  19. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  20. 2024Swing-by Dynamics in Concept Learning and Compositional Generalizationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Evolution of Concepts in Language Model Pre-Trainingcited by 8
  2. 2025Topological Invariance and Breakdown in Learningcited by 1
  3. 2026Mapping the Emergence of Regularization-Driven Dynamics in Grokkingcited by 1