# Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theory

arXiv: https://arxiv.org/abs/2307.04204 · HTML: https://mlsanity.com/paper/2307.04204/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 27 references that are in the map.
- First posted on arXiv 9 Jul 2023.

## Builds on (most cited first, 22 of 22)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [Implicit Regularization in Deep Matrix Factorization](https://mlsanity.com/paper/1905.13655/) (2019)
- [Implicit Regularization in Matrix Factorization](https://mlsanity.com/paper/1705.09280/) (2017)
- [Kernel and Rich Regimes in Overparametrized Models](https://mlsanity.com/paper/2002.09277/) (2020)
- [PyHessian: Neural Networks Through the Lens of the Hessian](https://mlsanity.com/paper/1912.07145/) (2019)
- [On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization](https://mlsanity.com/paper/1802.06509/) (2018)
- [Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss](https://mlsanity.com/paper/2002.04486/) (2020)
- [Implicit Bias of Gradient Descent on Linear Convolutional Networks](https://mlsanity.com/paper/1806.00468/) (2018)
- [Gradient descent aligns the layers of deep linear networks](https://mlsanity.com/paper/1810.02032/) (2018)
- [Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability](https://mlsanity.com/paper/2209.15594/) (2022)
- [Understanding Gradient Descent on Edge of Stability in Deep Learning](https://mlsanity.com/paper/2205.09745/) (2022)
- [Understanding the unstable convergence of gradient descent](https://mlsanity.com/paper/2204.01050/) (2022)
- [Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction](https://mlsanity.com/paper/2206.07085/) (2022)
- [On the Implicit Bias in Deep-Learning Algorithms](https://mlsanity.com/paper/2208.12591/) (2022)
- [A Unifying View on Implicit Bias in Training Linear Neural Networks](https://mlsanity.com/paper/2010.02501/) (2020)
- [Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stability](https://mlsanity.com/paper/2207.12678/) (2022)
- [Understanding Edge-of-Stability Training Dynamics with a Minimalist Example](https://mlsanity.com/paper/2210.03294/) (2022)
- [Learning threshold neurons via the "edge of stability"](https://mlsanity.com/paper/2212.07469/) (2022)
- [Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability](https://mlsanity.com/paper/2305.11788/) (2023)
- [Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyond](https://mlsanity.com/paper/2305.13064/) (2023)

## Built on by (most cited first, 26 of 26)

- [Does SGD really happen in tiny subspaces?](https://mlsanity.com/paper/2405.16002/) (2024)
- [Super Consistency of Neural Network Landscapes and Learning Rate Transfer](https://mlsanity.com/paper/2402.17457/) (2024)
- [Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos](https://mlsanity.com/paper/2311.02076/) (2023)
- [From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regression](https://mlsanity.com/paper/2310.01687/) (2023)
- [Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizes](https://mlsanity.com/paper/2406.05033/) (2024)
- [Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training](https://mlsanity.com/paper/2507.09846/) (2025)
- [Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stability](https://mlsanity.com/paper/2602.01480/) (2026)
- [Non-Euclidean Gradient Descent Operates at the Edge of Stability](https://mlsanity.com/paper/2603.05002/) (2026)
- [Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Region](https://mlsanity.com/paper/2509.25351/) (2025)
- [From Logistic Regression to the Perceptron Algorithm: Exploring Gradient Descent with Large Step Sizes](https://mlsanity.com/paper/2412.08424/) (2024)
- [Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and More](https://mlsanity.com/paper/2506.06940/) (2025)
- [Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapults](https://mlsanity.com/paper/2311.15051/) (2023)
- [On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentials](https://mlsanity.com/paper/2602.14789/) (2026)
- [Training on the Edge of Stability Is Caused by Layerwise Jacobian Alignment](https://mlsanity.com/paper/2406.00127/) (2024)
- [A Physics-Inspired Optimizer: Velocity Regularized Adam](https://mlsanity.com/paper/2505.13196/) (2025)
- [Product-Stability: Provable Convergence for Gradient Descent on the Edge of Stability](https://mlsanity.com/paper/2604.02653/) (2026)
- [Unbiased Gradients, Moving Stability Boundaries: Exact Mini-Batch Geometry in Linear Self-Attention](https://mlsanity.com/paper/2605.21292/) (2026)
- [The Road Taken: The Role of Optimizers at the Edge of Stability](https://mlsanity.com/paper/2608.18415/) (2026)
- [Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal](https://mlsanity.com/paper/2608.20638/) (2026)
- [Zeroth-Order Optimization at the Edge of Stability](https://mlsanity.com/paper/2604.14669/) (2026)
- [A Bifurcation Theory Framework for Gradient Descent on the Edge of Stability](https://mlsanity.com/paper/2606.15551/) (2026)
- [The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flow](https://mlsanity.com/paper/2609.01034/) (2026)
- [A Journey to the Edge of Stability](https://mlsanity.com/paper/2609.32290/) (2026)
- [Muon Sublates the Edge of Stability in LLM Pretraining](https://mlsanity.com/paper/2609.34915/) (2026)
- [How Does Local Landscape Geometry Evolve in Language Model Pre-Training?](https://mlsanity.com/paper/2609.39767/) (2026)
- [Non-Euclidean Gradient Descent Has Many Edges of Stability](https://mlsanity.com/paper/2610.15223/) (2026)

## Abstract (arXiv)

Cohen et al. (2021) empirically study the evolution of the largest eigenvalue of the loss Hessian, also known as sharpness, along the gradient descent (GD) trajectory and observe the Edge of Stability (EoS) phenomenon. The sharpness increases at the early phase of training (referred to as progressive sharpening), and eventually saturates close to the threshold of 2 / (step size). In this paper, we start by demonstrating through empirical studies that when the EoS phenomenon occurs, different GD trajectories (after a proper reparameterization) align on a specific bifurcation diagram independent of initialization. We then rigorously prove this trajectory alignment phenomenon for a two-layer fully-connected linear network and a single-neuron nonlinear network trained with a single data point. Our trajectory alignment analysis establishes both progressive sharpening and EoS phenomena, encompassing and extending recent findings in the literature.