mlsanityData to 7 Oct 2026

Non-Euclidean Gradient Descent Operates at the Edge of Stability

arXiv 2603.05002 ↗First posted 5 Mar 2026cs.LG math.OC stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

The Edge of Stability (EoS) is a phenomenon where the sharpness (largest eigenvalue) of the Hessian approaches and then hovers near the stability threshold 2/η during gradient descent (GD) with step size η. Despite (apparently) violating classical smoothness assumptions, EoS has been widely observed in deep learning, but its theoretical foundations remain incomplete. We provide an interpretation of EoS through the lens of Directional Smoothness [Mishkin et al., 2024]. This interpretation naturally extends to non-Euclidean norms, which we use to define generalized sharpness under an arbitrary norm. Our generalized sharpness measure includes previously studied vanilla GD and preconditioned GD as special cases, as well as methods for which EoS has not been studied, such as ℓ∞-descent, Block CD, Spectral GD, and their normalized versions. Through experiments on neural networks, we show that non-Euclidean GD with our generalized sharpness also exhibits progressive sharpening followed by oscillations around or above the threshold 2/η. Practically, our framework provides a geometry-aware spectral diagnostic that can be applied across a broad class of non-Euclidean gradient methods.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 29 of its 49 references that are in the map. First posted on arXiv 5 Mar 2026.

Builds on

29 of 49 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2021Convex optimizationcited by 2,479
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  6. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  7. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  8. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  9. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  10. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  11. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  12. 2024Old Optimizer, New Norm: An Anthologycited by 173
17 more
  1. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  2. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 137
  3. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  4. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 104
  5. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  6. 2016Convergence Rate of Frank-Wolfe for Non-Convex Objectivescited by 59
  7. 2024Understanding Optimization in Deep Learning with Central Flowscited by 57
  8. 2023Optimal Linear Decay Learning Rate Schedules and Further Refinementscited by 41
  9. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
  10. 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 24
  11. 2024Edge of Stochastic Stability: Revisiting the Edge of Stability for SGDcited by 24
  12. 2025Why Gradients Rapidly Increase Near the End of Trainingcited by 21
  13. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
  14. 2023Sharpness-Aware Minimization and the Edge of Stabilitycited by 16
  15. 2024Directional Smoothness and Gradient Methods: Convergence and Adaptivitycited by 15
  16. 2024Exploring the loss landscape of regularized neural networks via convex dualitycited by 11
  17. 2025Why Do We Need Warm-up? A Theoretical Perspectivecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Muon is Not That Special: Random or Inverted Spectra Work Just as Wellcited by 8
  2. 2026Does Weight Decay Enhance Training Stability?cited by 1
  3. 2026The Road Taken: The Role of Optimizers at the Edge of Stabilitycited by 1
  4. 2026Edge of Stability Selectively Shapes Learning Across the Data Distributioncited by 0
  5. 2026How the Hessian-Spectrum of Neural Networks Depends on Datacited by 0
  6. 2026A Journey to the Edge of Stabilitycited by 0
  7. 2026Muon Sublates the Edge of Stability in LLM Pretrainingcited by 0