mlsanityData to 14 Oct 2026

Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability

arXiv 2305.11788 ↗First posted 19 May 2023cs.LG stat.MLcited by 42

Not on the map. Why?

Abstract (arXiv)

Recent research has observed that in machine learning optimization, gradient descent (GD) often operates at the edge of stability (EoS) [Cohen, et al., 2021], where the stepsizes are set to be large, resulting in non-monotonic losses induced by the GD iterates. This paper studies the convergence and implicit bias of constant-stepsize GD for logistic regression on linearly separable data in the EoS regime. Despite the presence of local oscillations, we prove that the logistic loss can be minimized by GD with any constant stepsize over a long time scale. Furthermore, we prove that with any constant stepsize, the GD iterates tend to infinity when projected to a max-margin direction (the hard-margin SVM direction) and converge to a fixed vector that minimizes a strongly convex potential when projected to the orthogonal complement of the max-margin direction. In contrast, we also show that in the EoS regime, GD iterates may diverge catastrophically under the exponential loss, highlighting the superiority of the logistic loss. These theoretical findings are in line with numerical simulations and complement existing theories on the convergence and implicit bias of GD for logistic regression, which are only applicable when the stepsizes are sufficiently small.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Builds on 29 of its 41 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

29 of 41 references.

References · 29

  1. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  2. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  4. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  5. 2017Implicit Regularization in Matrix Factorizationcited by 322
  6. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  7. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
  8. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  9. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  10. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  11. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  12. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
17 more
  1. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  2. 2018Convergence of Gradient Descent on Separable Datacited by 110
  3. 2022Understanding the unstable convergence of gradient descentcited by 86
  4. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  5. 2018Risk and parameter convergence of logistic regressioncited by 85
  6. 2018A Walk with SGDcited by 78
  7. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  8. 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 64
  9. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 63
  10. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 56
  11. 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
  12. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  13. 2022Learning threshold neurons via the "edge of stability"cited by 51
  14. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 50
  15. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 43
  16. 2021Fast Margin Maximization via Dual Accelerationcited by 33
  17. 2020Stochasticity of Deterministic Gradient Descent: Large Learning Rate for Multiscale Objective Functioncited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 8; 2025: 12; 2026: 15.

42 papers.

Sort:

  1. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 80
  2. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  3. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 41
  4. 2024Does SGD really happen in tiny subspaces?cited by 36
  5. 2024The Implicit Bias of Adam on Separable Datacited by 36
  6. 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 35
  7. 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 26
  8. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 22
  9. 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 17
  10. 2024Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimizationcited by 17
  11. 2024Learning Associative Memories with Gradient Descentcited by 16
  12. 2024Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizescited by 14
30 more
  1. 2025Benefits of Early Stopping in Gradient Descent for Overparameterized Logistic Regressioncited by 9
  2. 2025Large Stepsizes Accelerate Gradient Descent for Regularized Logistic Regressioncited by 9
  3. 2023Benign Oscillation of Stochastic Gradient Descent with Large Learning Ratescited by 8
  4. 2025Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizescited by 7
  5. 2025Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and Morecited by 5
  6. 2025Gradient Descent on Logistic Regression: Do Large Step-Sizes Work with Data on the Sphere?cited by 5
  7. 2023Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescalingcited by 4
  8. 2025Convergence Rates for Gradient Descent on the Edge of Stability in Overparametrised Least Squarescited by 4
  9. 2023Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapultscited by 3
  10. 2026Understanding and inverse design of implicit bias in stochastic learning: a geometric perspectivecited by 3
  11. 2026Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learningcited by 3
  12. 2026Edge Flow: A Tractable and Predictive Continuous-Time Model for Gradient Descent at the Edge of Stabilitycited by 3
  13. 2025Variational Learning Finds Flatter Solutions at the Edge of Stabilitycited by 2
  14. 2025Constant Stepsize Local GD for Logistic Regression: Acceleration by Instabilitycited by 2
  15. 2025Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamicscited by 2
  16. 2024Training on the Edge of Stability Is Caused by Layerwise Jacobian Alignmentcited by 1
  17. 2025Any-stepsize Gradient Descent for Separable Data under Fenchel-Young Lossescited by 1
  18. 2026Tight Bounds for Logistic Regression with Large Stepsize Gradient Descent in Low Dimensioncited by 1
  19. 2026Product-Stability: Provable Convergence for Gradient Descent on the Edge of Stabilitycited by 1
  20. 2026Finite-Sample Performance of Gradient Descent in Logistic Regression with Gaussian Designcited by 1
  21. 2026Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minimacited by 1
  22. 2026The Road Taken: The Role of Optimizers at the Edge of Stabilitycited by 1
  23. 2025Convergence of optimizers implies eigenvalues filtering at equilibriumcited by 0
  24. 2026Over-Alignment vs Over-Fitting: The Role of Feature Learning Strength in Generalizationcited by 0
  25. 2026Exponential Convergence of (Stochastic) Gradient Descent for Separable Logistic Regressioncited by 0
  26. 2026Efficient Logistic Regression with Mixture of Sigmoidscited by 0
  27. 2026A Mechanism Study of Delayed Loss Spikes in Batch-Normalized Linear Modelscited by 0
  28. 2026Centre manifold theorem for maps along manifolds of fixed pointscited by 0
  29. 2026The Implicit Bias of Depth: From Neural Collapse to Softmax Codescited by 0
  30. 2026The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flowcited by 0