mlsanityData to 14 Oct 2026

Learning threshold neurons via the "edge of stability"

arXiv 2212.07469 ↗First posted 14 Dec 2022cs.LG cs.AI math.OCcited by 51

Not on the map. Why?

Abstract (arXiv)

Existing analyses of neural network training often operate under the unrealistic assumption of an extremely small learning rate. This lies in stark contrast to practical wisdom and empirical studies, such as the work of J. Cohen et al. (ICLR 2021), which exhibit startling new phenomena (the "edge of stability" or "unstable convergence") and potential benefits for generalization in the large learning rate regime. Despite a flurry of recent works on this topic, however, the latter effect is still poorly understood. In this paper, we take a step towards understanding genuinely non-convex training dynamics with large learning rates by performing a detailed analysis of gradient descent for simplified models of two-layer neural networks. For these models, we provably establish the edge of stability phenomenon and discover a sharp phase transition for the step size below which the neural network fails to learn "threshold-like" neurons (i.e., neurons with a non-zero first-layer bias). This elucidates one possible mechanism by which the edge of stability can in fact lead to better generalization, as threshold neurons are basic building blocks with useful inductive bias for many tasks.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Builds on 35 of its 62 references that are in the map. First posted on arXiv 14 Dec 2022.

Builds on

35 of 62 references.

References · 35

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  4. 2018On Lazy Training in Differentiable Programmingcited by 675
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  6. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  7. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  8. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  9. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  10. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  11. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  12. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
23 more
  1. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  2. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  3. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  4. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  5. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  6. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 106
  7. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 106
  8. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 88
  9. 2022Understanding the unstable convergence of gradient descentcited by 86
  10. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  11. 2018A Walk with SGDcited by 78
  12. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  13. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 63
  14. 2020Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalizationcited by 62
  15. 2022Mean-Field Langevin Dynamics: Exponential Convergence and Annealingcited by 61
  16. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 56
  17. 2022Convex Analysis of the Mean Field Langevin Dynamicscited by 55
  18. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 50
  19. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 43
  20. 2022Beyond the Edge of Stability via Two-step Gradient Updatescited by 38
  21. 2015Simple, Efficient, and Neural Algorithms for Sparse Codingcited by 37
  22. 2020Learning Rate Annealing Can Provably Help Generalization, Even for Convex Problemscited by 11
  23. 2022On the Benefits of Large Learning Rates for Kernel Methodscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 17; 2024: 11; 2025: 13; 2026: 10.

51 papers.

Sort:

By area:

  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  2. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 98
  3. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  4. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 36
  5. 2024Does SGD really happen in tiny subspaces?cited by 36
  6. 2023Why Does Sharpness-Aware Minimization Generalize Better Than SGD?cited by 28
  7. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26
  8. 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 26
  9. 2023Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapultcited by 25
  10. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 22
  11. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 21
  12. 2023When and Why Momentum Accelerates SGD:An Empirical Studycited by 19
38 more
  1. 2023Sharpness-Aware Minimization and the Edge of Stabilitycited by 17
  2. 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 17
  3. 2024Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learningcited by 17
  4. 2024Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimizationcited by 17
  5. 2023Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networkscited by 15
  6. 2024A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practicescited by 15
  7. 2024Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizescited by 14
  8. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 14
  9. 2023How to escape sharp minima with random perturbationscited by 13
  10. 2023Convergence of Alternating Gradient Descent for Matrix Factorizationcited by 12
  11. 2024Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizescited by 12
  12. 2024Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problemscited by 10
  13. 2025Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Trainingcited by 9
  14. 2026Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stabilitycited by 9
  15. 2025Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizescited by 7
  16. 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 7
  17. 2024Where Do Large Learning Rates Lead Us?cited by 6
  18. 2025Zeroth-Order Optimization Finds Flat Minimacited by 6
  19. 2025Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and Morecited by 5
  20. 2023Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapultscited by 3
  21. 2026Edge Flow: A Tractable and Predictive Continuous-Time Model for Gradient Descent at the Edge of Stabilitycited by 3
  22. 2024GD doesn't make the cut: Three ways that non-differentiability affects neural network trainingcited by 2
  23. 2025Gradient Descent Converges Linearly to Flatter Minima than Gradient Flow in Shallow Linear Networkscited by 2
  24. 2025Understanding Nonlinear Implicit Bias via Region Counts in Input Spacecited by 2
  25. 2025Variational Learning Finds Flatter Solutions at the Edge of Stabilitycited by 2
  26. 2025Structured Transformations for Stable and Interpretable Neural Computationcited by 2
  27. 2026Generalization at the Edge of Stabilitycited by 2
  28. 2024Training on the Edge of Stability Is Caused by Layerwise Jacobian Alignmentcited by 1
  29. 2025Any-stepsize Gradient Descent for Separable Data under Fenchel-Young Lossescited by 1
  30. 2025Deep Edge Filter: Return of the Human-Crafted Layer in Deep Learningcited by 1
  31. 2026Does SGD Seek Flatness or Sharpness? An Exactly Solvable Modelcited by 1
  32. 2026Tight Bounds for Logistic Regression with Large Stepsize Gradient Descent in Low Dimensioncited by 1
  33. 2026Product-Stability: Provable Convergence for Gradient Descent on the Edge of Stabilitycited by 1
  34. 2026The Road Taken: The Role of Optimizers at the Edge of Stabilitycited by 1
  35. 2026Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversalcited by 1
  36. 2024Criteria and Bias of Parameterized Linear Regression under Edge of Stability Regimecited by 0
  37. 2025Technical Debt in In-Context Learning: Diminishing Efficiency in Long Contextcited by 0
  38. 2026Edge of Stability Selectively Shapes Learning Across the Data Distributioncited by 0

50 most cited of 51 All 51 (JSON)