mlsanityData to 14 Oct 2026

Edge of Stability Selectively Shapes Learning Across the Data Distribution

arXiv 2606.04212 ↗First posted 2 Jun 2026cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Existing analyses of the edge of stability (EoS) treat it as a global property of optimization. We show that it is also selective: the stability constraint redistributes learning across subsets of the training distribution, amplifying progress on some groups while suppressing progress on others. Using a branching intervention that enters or exits the EoS regime from the same training state, we causally demonstrate this trade-off and identify two necessary conditions for a group to benefit. First, its aggregate gradient must align with the top Hessian eigenvector. We isolate this mechanism with a controlled perturbation that preserves distance but randomizes direction, destroying alignment and eliminating the advantage. Second, the group must sustain non-vanishing gradient magnitude over time. Under cross-entropy loss, gradient saturation decouples confidently classified groups, shifting the advantage to output-outliers, whose gradients persist. Together, these results show that EoS functions not only as a stability boundary, but as a mechanism governing the allocation of learning across the data distribution.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 43 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

38 of 43 references.

References · 38

  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  2. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  6. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  7. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  8. 2017Exploring Generalization in Deep Learningcited by 600
  9. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  10. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  11. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  12. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
26 more
  1. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  2. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
  3. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  4. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  5. 2017Three Factors Influencing Minima in SGDcited by 273
  6. 2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 267
  7. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  8. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  9. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  10. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  11. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 101
  12. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 88
  13. 2022Understanding the unstable convergence of gradient descentcited by 86
  14. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  15. 2017Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapescited by 84
  16. 2018A Walk with SGDcited by 78
  17. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 56
  18. 2022Learning threshold neurons via the "edge of stability"cited by 51
  19. 2022Beyond the Edge of Stability via Two-step Gradient Updatescited by 38
  20. 2024Edge of Stochastic Stability: Revisiting the Edge of Stability for SGDcited by 30
  21. 2023Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimizationcited by 13
  22. 2026Momentum Further Constrains Sharpness at the Edge of Stochastic Stabilitycited by 10
  23. 2026Non-Euclidean Gradient Descent Operates at the Edge of Stabilitycited by 9
  24. 2026A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMscited by 8
  25. 2026Too Sharp, Too Sure: When Calibration Follows Curvaturecited by 3
  26. 2026Does Weight Decay Enhance Training Stability?cited by 1

Built on by

Not cited yet.