mlsanityData to 7 Oct 2026

Second-order regression models exhibit progressive sharpening to the edge of stability

arXiv 2210.04860 ↗First posted 10 Oct 2022cs.LG cs.AI math.OCcited by 40

Not on the map. Why?

Abstract (arXiv)

Recent studies of gradient descent with large step sizes have shown that there is often a regime with an initial increase in the largest eigenvalue of the loss Hessian (progressive sharpening), followed by a stabilization of the eigenvalue near the maximum value which allows convergence (edge of stability). These phenomena are intrinsically non-linear and do not happen for models in the constant Neural Tangent Kernel (NTK) regime, for which the predictive function is approximately linear in the parameters. As such, we consider the next simplest class of predictive models, namely those that are quadratic in the parameters, which we call second-order regression models. For quadratic objectives in two dimensions, we prove that this second-order regression model exhibits progressive sharpening of the NTK eigenvalue towards a value that differs slightly from the edge of stability, which we explicitly compute. In higher dimensions, the model generically shows similar behavior, even without the specific structure of a neural network, suggesting that progressive sharpening and edge-of-stability behavior aren't unique features of neural networks, and could be a more general property of discrete learning algorithms in high-dimensional non-linear models.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 24 of its 28 references that are in the map. First posted on arXiv 10 Oct 2022.

Builds on

24 of 28 references.

References · 24

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  5. 2017Exploring Generalization in Deep Learningcited by 595
  6. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  7. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  8. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  9. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  10. 2021The Principles of Deep Learning Theorycited by 169
  11. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  12. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
12 more
  1. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  2. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  3. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  4. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  5. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  6. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  7. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
  8. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 54
  9. 2020Multiple Descent: Design Your Own Generalization Curvecited by 44
  10. 2019The asymptotic spectrum of the Hessian of DNN throughout trainingcited by 19
  11. 2022Quadratic models for understanding catapult dynamics of neural networkscited by 17
  12. 2019At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?cited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 11; 2025: 7; 2026: 9.

40 papers.

Sort:

By area:

  1. 2024Normalization and effective learning rates in reinforcement learningcited by 55
  2. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
  3. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
  4. 2023Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index modelscited by 25
  5. 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenoncited by 24
  6. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  7. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 21
  8. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 20
  9. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
  10. 2024Deep linear networks for regression are implicitly regularized towards flat minimacited by 17
  11. 2023Sharpness-Aware Minimization and the Edge of Stabilitycited by 16
  12. 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 16
28 more
  1. 2023Loss Spike in Training Neural Networkscited by 15
  2. 2023How connectivity structure shapes rich and lazy learning in neural circuitscited by 15
  3. 2024High dimensional analysis reveals conservative sharpening and a stochastic edge of stabilitycited by 15
  4. 2024Learning Associative Memories with Gradient Descentcited by 14
  5. 2024The Optimization Landscape of SGD Across the Feature Learning Strengthcited by 13
  6. 2024Neglected Hessian component explains mysteries in Sharpness regularizationcited by 12
  7. 2024Feature learning as alignment: a structural property of gradient descent in non-linear neural networkscited by 9
  8. 2023Catapult Dynamics and Phase Transitions in Quadratic Netscited by 8
  9. 2024Stepping on the Edge: Curvature Aware Learning Rate Tunerscited by 8
  10. 2025What Can Grokking Teach Us About Learning Under Nonstationarity?cited by 7
  11. 2026Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stabilitycited by 6
  12. 2026The Origin of Edge of Stabilitycited by 6
  13. 2024To Clip or not to Clip: the Dynamics of SGD with Gradient Clipping in High-Dimensionscited by 5
  14. 2025Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and Morecited by 5
  15. 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 5
  16. 2023On the Interplay Between Stepsize Tuning and Progressive Sharpeningcited by 4
  17. 2025A Minimalist Example of Edge-of-Stability and Progressive Sharpeningcited by 4
  18. 2025Convergence Rates for Gradient Descent on the Edge of Stability in Overparametrised Least Squarescited by 3
  19. 2023Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapultscited by 2
  20. 2025Variational Learning Finds Flatter Solutions at the Edge of Stabilitycited by 2
  21. 2026Does SGD Seek Flatness or Sharpness? An Exactly Solvable Modelcited by 1
  22. 2026SGD at the Edge of Stability: The Stochastic Sharpness Gapcited by 1
  23. 2026Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minimacited by 1
  24. 2026The Road Taken: The Role of Optimizers at the Edge of Stabilitycited by 1
  25. 2025A Special Case of Quadratic Extrapolation Under the Neural Tangent Kernelcited by 0
  26. 2026Centre manifold theorem for maps along manifolds of fixed pointscited by 0
  27. 2026The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flowcited by 0
  28. 2026A Journey to the Edge of Stabilitycited by 0