mlsanityData to 7 Oct 2026

Training on the Edge of Stability Is Caused by Layerwise Jacobian Alignment

arXiv 2406.00127 ↗First posted 31 May 2024stat.ML cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

During neural network training, the sharpness of the Hessian matrix of the training loss rises until training is on the edge of stability. As a result, even nonstochastic gradient descent does not accurately model the underlying dynamical system defined by the gradient flow of the training loss. We use an exponential Euler solver to train the network without entering the edge of stability, so that we accurately approximate the true gradient descent dynamics. We demonstrate experimentally that the increase in the sharpness of the Hessian matrix is caused by the layerwise Jacobian matrices of the network becoming aligned, so that a small change in the network preactivations near the inputs of the network can cause a large change in the outputs of the network. We further demonstrate that the degree of alignment scales with the size of the dataset by a power law with a coefficient of determination between 0.74 and 0.98.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 24 of its 41 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

24 of 41 references.

References · 24

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  6. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  7. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  8. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  9. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  10. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  11. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  12. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
12 more
  1. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  2. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  3. 2022Understanding the unstable convergence of gradient descentcited by 84
  4. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  5. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
  6. 2022Learning threshold neurons via the "edge of stability"cited by 50
  7. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 42
  8. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  9. 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 34
  10. 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 24
  11. 2023Sharpness-Aware Minimization and the Edge of Stabilitycited by 16
  12. 2023Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimizationcited by 13

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025PreMoE: Proactive Inference for Efficient Mixture-of-Expertscited by 1