mlsanityData to 7 Oct 2026

Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Data

arXiv 2607.06013 ↗First posted 7 Jul 2026cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

Adaptive gradient methods can favor max-margin separators that differ from gradient descent, yet a fixed positive numerical stability constant eventually changes the update geometry again. This paper studies the rate-controlled middle case for full-batch linear classification on separable data. For memoryless stability-annealed smoothed-sign descent with weighted exponential loss, we prove that the normalized iterates converge to the minimizer of a convex Burg-type barrier over a margin slice. The proof rewrites the dynamics exactly as entropic mirror ascent on a concave dual objective, controls the dual gap by a KL recursion, and yields an explicit S_t^{-1/2} normalized-iterate envelope. The static barrier geometry is fully characterized, including KKT conditions and both endpoint limits. Experiments validate the exact dual identities to floating-point error, illustrate the predicted path and rate diagram, and show an empirical fixed-epsilon crossover scaling in cumulative time. We further report robustness and boundary diagnostics for logistic tails, fixed-epsilon crossover, and adaptive-method variants, delineating the scope of the proved smoothed-sign theory.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 25 references that are in the map. First posted on arXiv 7 Jul 2026.

Builds on

19 of 25 references.

References · 19

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019On the Convergence of Adam and Beyondcited by 834
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  4. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  5. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  6. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  7. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  8. 2020A Simple Convergence Proof of Adam and Adagradcited by 169
  9. 2018Convergence of Gradient Descent on Separable Datacited by 108
  10. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  11. 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 63
  12. 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
7 more
  1. 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 43
  2. 2013Margins, Shrinkage, and Boostingcited by 35
  3. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  4. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 33
  5. 2022Mirror Descent Maximizes Generalized Margin and Can Be Implemented Efficientlycited by 25
  6. 2023On the Implicit Bias of Adamcited by 25
  7. 2019The Implicit Bias of AdaGrad on Separable Datacited by 15

Built on by

Not cited yet.