mlsanityData to 7 Oct 2026

A Minimalist Example of Edge-of-Stability and Progressive Sharpening

arXiv 2503.02809 ↗First posted 4 Mar 2025cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Recent advances in deep learning optimization have unveiled two intriguing phenomena under large learning rates: Edge of Stability (EoS) and Progressive Sharpening (PS), challenging classical Gradient Descent (GD) analyses. Current research approaches, using either generalist frameworks or minimalist examples, face significant limitations in explaining these phenomena. This paper advances the minimalist approach by introducing a two-layer network with a two-dimensional input, where one dimension is relevant to the response and the other is irrelevant. Through this model, we rigorously prove the existence of progressive sharpening and self-stabilization under large learning rates, and establish non-asymptotic analysis of the training dynamics and sharpness along the entire GD trajectory. Besides, we connect our minimalist example to existing works by reconciling the existence of a well-behaved ``stable set" between minimalist and generalist analyses, and extending the analysis of Gradient Flow Solution sharpness to our two-dimensional input scenario. These findings provide new insights into the EoS phenomenon from both parameter and input data distribution perspectives, potentially informing more effective optimization strategies in deep learning practice.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 19 of its 25 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

19 of 25 references.

References · 19

  1. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  2. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  3. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  4. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  5. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  6. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  7. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  8. 2022Understanding the unstable convergence of gradient descentcited by 84
  9. 2018A Walk with SGDcited by 74
  10. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  11. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
  12. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 54
7 more
  1. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 48
  2. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  3. 2022Beyond the Edge of Stability via Two-step Gradient Updatescited by 38
  4. 2022Robust Training of Neural Networks Using Scale Invariant Architecturescited by 29
  5. 2023Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapultcited by 23
  6. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 20
  7. 2023Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimizationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Convergence Rates for Gradient Descent on the Edge of Stability in Overparametrised Least Squarescited by 3
  2. 2026Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minimacited by 1
  3. 2026Centre manifold theorem for maps along manifolds of fixed pointscited by 0
  4. 2026The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flowcited by 0