mlsanityData to 7 Oct 2026

High dimensional analysis reveals conservative sharpening and a stochastic edge of stability

arXiv 2404.19261 ↗First posted 30 Apr 2024cs.LG math.OC math.STcited by 15

Not on the map. Why?

Abstract (arXiv)

Recent empirical and theoretical work has shown that the dynamics of the large eigenvalues of the training loss Hessian have some remarkably robust features across models and datasets in the full batch regime. There is often an early period of progressive sharpening where the large eigenvalues increase, followed by stabilization at a predictable value known as the edge of stability. Previous work showed that in the stochastic setting, the eigenvalues increase more slowly - a phenomenon we call conservative sharpening. We provide a theoretical analysis of a simple high-dimensional model which shows the origin of this slowdown. We also show that there is an alternative stochastic edge of stability which arises at small batch size that is sensitive to the trace of the Neural Tangent Kernel rather than the large Hessian eigenvalues. We conduct an experimental study which highlights the qualitative differences from the full batch phenomenology, and suggests that controlling the stochastic edge of stability can help optimization.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 30 of its 33 references that are in the map. First posted on arXiv 30 Apr 2024.

Builds on

30 of 33 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  7. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  8. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  9. 2017Three Factors Influencing Minima in SGDcited by 270
  10. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  11. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  12. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
18 more
  1. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  2. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  3. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  4. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
  5. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  6. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 61
  7. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  8. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 42
  9. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  10. 2021SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticalitycited by 33
  11. 2022Homogenization of SGD in high-dimensions: Exact dynamics and generalization propertiescited by 26
  12. 2023Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index modelscited by 25
  13. 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenoncited by 24
  14. 2022Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensionscited by 23
  15. 2022Quadratic models for understanding catapult dynamics of neural networkscited by 17
  16. 2024Neglected Hessian component explains mysteries in Sharpness regularizationcited by 12
  17. 2023Exact Mean Square Linear Stability Analysis for SGDcited by 10
  18. 2023On the Interplay Between Stepsize Tuning and Progressive Sharpeningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 7.

15 papers.

Sort:

  1. 2024Does SGD really happen in tiny subspaces?cited by 35
  2. 2025Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networkscited by 21
  3. 2024Stepping on the Edge: Curvature Aware Learning Rate Tunerscited by 8
  4. 2026A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMscited by 8
  5. 2026Momentum Further Constrains Sharpness at the Edge of Stochastic Stabilitycited by 8
  6. 2024To Clip or not to Clip: the Dynamics of SGD with Gradient Clipping in High-Dimensionscited by 5
  7. 2024Exact Risk Curves of signSGD in High-Dimensions: Quantifying Preconditioning and Noise-Compression Effectscited by 5
  8. 2025Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and Morecited by 5
  9. 2025Per-example gradients: a new frontier for understanding and improving optimizerscited by 2
  10. 2026High dimensional theory of two-phase optimizerscited by 1
  11. 2026Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regimecited by 1
  12. 2025Convergence of optimizers implies eigenvalues filtering at equilibriumcited by 0
3 more
  1. 2026Large Spikes in Stochastic Gradient Descent: A Large-Deviations Viewcited by 0
  2. 2026What do near-optimal learning rate schedules look like?cited by 0
  3. 2026Unbiased Gradients, Moving Stability Boundaries: Exact Mini-Batch Geometry in Linear Self-Attentioncited by 0