mlsanityData to 14 Oct 2026

Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theory

arXiv 2307.04204 ↗First posted 9 Jul 2023cs.LG math.OC stat.MLcited by 26

Not on the map. Why?

Abstract (arXiv)

Cohen et al. (2021) empirically study the evolution of the largest eigenvalue of the loss Hessian, also known as sharpness, along the gradient descent (GD) trajectory and observe the Edge of Stability (EoS) phenomenon. The sharpness increases at the early phase of training (referred to as progressive sharpening), and eventually saturates close to the threshold of 2 / (step size). In this paper, we start by demonstrating through empirical studies that when the EoS phenomenon occurs, different GD trajectories (after a proper reparameterization) align on a specific bifurcation diagram independent of initialization. We then rigorously prove this trajectory alignment phenomenon for a two-layer fully-connected linear network and a single-neuron nonlinear network trained with a single data point. Our trajectory alignment analysis establishes both progressive sharpening and EoS phenomena, encompassing and extending recent findings in the literature.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 22 of its 27 references that are in the map. First posted on arXiv 9 Jul 2023.

Builds on

22 of 27 references.

References · 22

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  3. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  4. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  5. 2017Implicit Regularization in Matrix Factorizationcited by 322
  6. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  7. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  8. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  9. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  10. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  11. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  12. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
10 more
  1. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  2. 2022Understanding the unstable convergence of gradient descentcited by 86
  3. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  4. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  5. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 75
  6. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 63
  7. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 56
  8. 2022Learning threshold neurons via the "edge of stability"cited by 51
  9. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  10. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 4; 2026: 14.

26 papers.

Sort:

  1. 2024Does SGD really happen in tiny subspaces?cited by 36
  2. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  3. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 22
  4. 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 17
  5. 2024Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizescited by 14
  6. 2025Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Trainingcited by 9
  7. 2026Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stabilitycited by 9
  8. 2026Non-Euclidean Gradient Descent Operates at the Edge of Stabilitycited by 9
  9. 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 7
  10. 2024From Logistic Regression to the Perceptron Algorithm: Exploring Gradient Descent with Large Step Sizescited by 6
  11. 2025Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and Morecited by 5
  12. 2023Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapultscited by 3
14 more
  1. 2026On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentialscited by 3
  2. 2024Training on the Edge of Stability Is Caused by Layerwise Jacobian Alignmentcited by 1
  3. 2025A Physics-Inspired Optimizer: Velocity Regularized Adamcited by 1
  4. 2026Product-Stability: Provable Convergence for Gradient Descent on the Edge of Stabilitycited by 1
  5. 2026Unbiased Gradients, Moving Stability Boundaries: Exact Mini-Batch Geometry in Linear Self-Attentioncited by 1
  6. 2026The Road Taken: The Role of Optimizers at the Edge of Stabilitycited by 1
  7. 2026Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversalcited by 1
  8. 2026Zeroth-Order Optimization at the Edge of Stabilitycited by 0
  9. 2026A Bifurcation Theory Framework for Gradient Descent on the Edge of Stabilitycited by 0
  10. 2026The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flowcited by 0
  11. 2026A Journey to the Edge of Stabilitycited by 0
  12. 2026Muon Sublates the Edge of Stability in LLM Pretrainingcited by 0
  13. 2026How Does Local Landscape Geometry Evolve in Language Model Pre-Training?cited by 0
  14. 2026Non-Euclidean Gradient Descent Has Many Edges of Stabilitycited by 0