mlsanityData to 7 Oct 2026

Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos

arXiv 2311.02076 ↗First posted 3 Nov 2023cs.LG cond-mat.dis-nn nlin.CDcited by 21

Not on the map. Why?

Abstract (arXiv)

In gradient descent dynamics of neural networks, the top eigenvalue of the loss Hessian (sharpness) displays a variety of robust phenomena throughout training. This includes early time regimes where the sharpness may decrease during early periods of training (sharpness reduction), and later time behavior such as progressive sharpening and edge of stability. We demonstrate that a simple 2-layer linear network (UV model) trained on a single training example exhibits all of the essential sharpness phenomenology observed in real-world scenarios. By analyzing the structure of dynamical fixed points in function space and the vector field of function updates, we uncover the underlying mechanisms behind these sharpness trends. Our analysis reveals (i) the mechanism behind early sharpness reduction and progressive sharpening, (ii) the required conditions for edge of stability, (iii) the crucial role of initialization and parameterization, and (iv) a period-doubling route to chaos on the edge of stability manifold as learning rate is increased. Finally, we demonstrate that various predictions from this simplified model generalize to real-world scenarios and discuss its limitations.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 37 of its 50 references that are in the map. First posted on arXiv 3 Nov 2023.

Builds on

37 of 50 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  4. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  7. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  8. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  9. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  10. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  11. 2021The Principles of Deep Learning Theorycited by 169
  12. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
25 more
  1. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  2. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  3. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  4. 2022Understanding the unstable convergence of gradient descentcited by 84
  5. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  6. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
  7. 2020Neural Kernels Without Tangentscited by 56
  8. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 54
  9. 2022Learning threshold neurons via the "edge of stability"cited by 50
  10. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 48
  11. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 42
  12. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
  13. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  14. 2020On the infinite width limit of neural networks with a standard parameterizationcited by 38
  15. 2022Beyond the Edge of Stability via Two-step Gradient Updatescited by 38
  16. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
  17. 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 24
  18. 2023Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapultcited by 23
  19. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 20
  20. 2022Meta-Principled Family of Hyperparameter Scaling Strategiescited by 18
  21. 2022Quadratic models for understanding catapult dynamics of neural networkscited by 17
  22. 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 16
  23. 2023On a continuous time model of gradient descent dynamics and instability in deep learningcited by 11
  24. 2021Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applicationscited by 8
  25. 2023Catapult Dynamics and Phase Transitions in Quadratic Netscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 6; 2026: 10.

21 papers.

Sort:

  1. 2024Why Warmup the Learning Rate? Underlying Mechanisms and Improvementscited by 45
  2. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  3. 2024The Optimization Landscape of SGD Across the Feature Learning Strengthcited by 13
  4. 2024Stepping on the Edge: Curvature Aware Learning Rate Tunerscited by 8
  5. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 8
  6. 2026A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMscited by 8
  7. 2026A Defense of the Quadratic Modelcited by 6
  8. 2025Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and Morecited by 5
  9. 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 5
  10. 2024Three Mechanisms of Feature Learning in a Linear Networkcited by 3
  11. 2025Convergence Rates for Gradient Descent on the Edge of Stability in Overparametrised Least Squarescited by 3
  12. 2025Variational Learning Finds Flatter Solutions at the Edge of Stabilitycited by 2
9 more
  1. 2026SGD at the Edge of Stability: The Stochastic Sharpness Gapcited by 1
  2. 2026Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Ratecited by 1
  3. 2026Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minimacited by 1
  4. 2025Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMscited by 0
  5. 2026Centre manifold theorem for maps along manifolds of fixed pointscited by 0
  6. 2026Generalization at the Edge of Stabilitycited by 0
  7. 2026How the Hessian-Spectrum of Neural Networks Depends on Datacited by 0
  8. 2026The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flowcited by 0
  9. 2026A Journey to the Edge of Stabilitycited by 0