mlsanityData to 14 Oct 2026

Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stability

arXiv 2207.12678 ↗First posted 26 Jul 2022cs.LGcited by 63

Not on the map. Why?

Abstract (arXiv)

Recent findings (e.g., arXiv:2103.00065) demonstrate that modern neural networks trained by full-batch gradient descent typically enter a regime called Edge of Stability (EOS). In this regime, the sharpness, i.e., the maximum Hessian eigenvalue, first increases to the value 2/(step size) (the progressive sharpening phase) and then oscillates around this value (the EOS phase). This paper aims to analyze the GD dynamics and the sharpness along the optimization trajectory. Our analysis naturally divides the GD trajectory into four phases depending on the change of the sharpness. We empirically identify the norm of output layer weight as an interesting indicator of sharpness dynamics. Based on this empirical observation, we attempt to theoretically and empirically explain the dynamics of various key quantities that lead to the change of sharpness in each phase of EOS. Moreover, based on certain assumptions, we provide a theoretical proof of the sharpness behavior in EOS regime in two-layer fully-connected linear neural networks. We also discuss some other empirical findings and the limitation of our theoretical results.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Builds on 26 of its 37 references that are in the map. First posted on arXiv 26 Jul 2022.

Builds on

26 of 37 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  5. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  6. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  7. 2018On Lazy Training in Differentiable Programmingcited by 675
  8. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  9. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  10. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  11. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  12. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
14 more
  1. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  2. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 159
  3. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  4. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  5. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 121
  6. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  7. 2022Understanding the unstable convergence of gradient descentcited by 86
  8. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  9. 2018A Walk with SGDcited by 78
  10. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  11. 2020The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networkscited by 58
  12. 2020Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networkscited by 45
  13. 2019Pathological spectra of the Fisher information metric and its variants in deep neural networkscited by 32
  14. 2019The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networkscited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 18; 2024: 9; 2025: 12; 2026: 18.

63 papers.

Sort:

By area:

  1. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  2. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 56
  3. 2022Learning threshold neurons via the "edge of stability"cited by 51
  4. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  5. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  6. 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 35
  7. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 31
  8. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26
  9. 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 26
  10. 2023Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapultcited by 25
  11. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 22
  12. 2024Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannotcited by 22
38 more
  1. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 21
  2. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
  3. 2023When and Why Momentum Accelerates SGD:An Empirical Studycited by 19
  4. 2024Deep linear networks for regression are implicitly regularized towards flat minimacited by 17
  5. 2024Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimizationcited by 17
  6. 2023Loss Spike in Training Neural Networkscited by 16
  7. 2023Irreducible Curriculum for Language Model Pretrainingcited by 16
  8. 2022Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspectivecited by 15
  9. 2023Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networkscited by 15
  10. 2023Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimizationcited by 13
  11. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  12. 2025Adaptive Preconditioners Trigger Loss Spikes in Adamcited by 13
  13. 2022Maximal Initial Learning Rates in Deep ReLU Networkscited by 11
  14. 2025Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Trainingcited by 9
  15. 2023On progressive sharpening, flat minima and generalisationcited by 7
  16. 2025Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizescited by 7
  17. 2023The Inductive Bias of Flatness Regularization for Deep Matrix Factorizationcited by 6
  18. 2025Zeroth-Order Optimization Finds Flat Minimacited by 6
  19. 2025Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stabilitycited by 6
  20. 2025Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and Morecited by 5
  21. 2024Deconstructing the Goldilocks Zone of Neural Network Initializationcited by 4
  22. 2024Simplicity Bias via Global Convergence of Sharpness Minimizationcited by 4
  23. 2025A Minimalist Example of Edge-of-Stability and Progressive Sharpeningcited by 4
  24. 2025Convergence Rates for Gradient Descent on the Edge of Stability in Overparametrised Least Squarescited by 4
  25. 2023Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapultscited by 3
  26. 2025New Evidence of the Two-Phase Learning Dynamics of Neural Networkscited by 3
  27. 2026On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentialscited by 3
  28. 2024GD doesn't make the cut: Three ways that non-differentiability affects neural network trainingcited by 2
  29. 2025Large Learning Rates Simultaneously Achieve Robustness to Spurious Correlations and Compressibilitycited by 2
  30. 2026Generalization at the Edge of Stabilitycited by 2
  31. 2026Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspectivecited by 2
  32. 2024Training on the Edge of Stability Is Caused by Layerwise Jacobian Alignmentcited by 1
  33. 2025A Physics-Inspired Optimizer: Velocity Regularized Adamcited by 1
  34. 2026Does SGD Seek Flatness or Sharpness? An Exactly Solvable Modelcited by 1
  35. 2026SGD at the Edge of Stability: The Stochastic Sharpness Gapcited by 1
  36. 2026Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minimacited by 1
  37. 2026Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decaycited by 1
  38. 2026The Road Taken: The Role of Optimizers at the Edge of Stabilitycited by 1

50 most cited of 63 All 63 (JSON)