mlsanityData to 7 Oct 2026

Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyond

arXiv 2305.13064 ↗First posted 22 May 2023cs.LG math.OC stat.MLcited by 20

Not on the map. Why?

Abstract (arXiv)

Recent research shows that when Gradient Descent (GD) is applied to neural networks, the loss almost never decreases monotonically. Instead, the loss oscillates as gradient descent converges to its ''Edge of Stability'' (EoS). Here, we find a quantity that does decrease monotonically throughout GD training: the sharpness attained by the gradient flow solution (GFS)-the solution that would be obtained if, from now until convergence, we train with an infinitesimal step size. Theoretically, we analyze scalar neural networks with the squared loss, perhaps the simplest setting where the EoS phenomena still occur. In this model, we prove that the GFS sharpness decreases monotonically. Using this result, we characterize settings where GD provably converges to the EoS in scalar networks. Empirically, we show that GD monotonically decreases the GFS sharpness in a squared regression model as well as practical neural network architectures.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 23 of its 35 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

23 of 35 references.

References · 23

  1. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  2. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  3. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  4. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  5. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  6. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  7. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  8. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  9. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  10. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  11. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  12. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
11 more
  1. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  2. 2022Understanding the unstable convergence of gradient descentcited by 84
  3. 2018A Walk with SGDcited by 74
  4. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  5. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  6. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
  7. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 54
  8. 2022Learning threshold neurons via the "edge of stability"cited by 50
  9. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 48
  10. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  11. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 5; 2025: 5; 2026: 5.

20 papers.

Sort:

  1. 2024Does SGD really happen in tiny subspaces?cited by 35
  2. 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 34
  3. 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 24
  4. 2023Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapultcited by 23
  5. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 21
  6. 2024Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimizationcited by 17
  7. 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 16
  8. 2024Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizescited by 12
  9. 2024Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizescited by 9
  10. 2025Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizescited by 5
  11. 2025Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and Morecited by 5
  12. 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 5
8 more
  1. 2025A Minimalist Example of Edge-of-Stability and Progressive Sharpeningcited by 4
  2. 2023Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapultscited by 2
  3. 2025Variational Learning Finds Flatter Solutions at the Edge of Stabilitycited by 2
  4. 2026On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentialscited by 2
  5. 2026SGD at the Edge of Stability: The Stochastic Sharpness Gapcited by 1
  6. 2026The Road Taken: The Role of Optimizers at the Edge of Stabilitycited by 1
  7. 2026Centre manifold theorem for maps along manifolds of fixed pointscited by 0
  8. 2026The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flowcited by 0