# On the Two Faces of Adam in Separable Linear Classification

arXiv: https://arxiv.org/abs/2609.33904 · HTML: https://mlsanity.com/paper/2609.33904/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 32 references that are in the map.
- First posted on arXiv 27 Sep 2026.

## Builds on (most cited first, 10 of 10)

- [Gradient Descent Maximizes the Margin of Homogeneous Neural Networks](https://mlsanity.com/paper/1906.05890/) (2019)
- [Old Optimizer, New Norm: An Anthology](https://mlsanity.com/paper/2409.20325/) (2024)
- [Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning](https://mlsanity.com/paper/2012.09839/) (2020)
- [Risk and parameter convergence of logistic regression](https://mlsanity.com/paper/1803.07300/) (2018)
- [Transformers as Support Vector Machines](https://mlsanity.com/paper/2308.16898/) (2023)
- [Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimization](https://mlsanity.com/paper/2404.04454/) (2024)
- [Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks](https://mlsanity.com/paper/2410.22069/) (2024)
- [The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networks](https://mlsanity.com/paper/2602.16340/) (2026)
- [The Implicit Bias of Steepest Descent with Mini-batch Stochastic Gradient](https://mlsanity.com/paper/2602.11557/) (2026)
- [Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Data](https://mlsanity.com/paper/2607.06013/) (2026)

## Abstract (arXiv)

We consider the behavior of deterministic, full-batch, bias-corrected Adam in separable linear classification with softmax parametrization under log-loss. In this setting, under a wide range of conditions Adam is known to approach max-norm-margin optimality when its stability constant ε is zero, while with a positive ε, it is known to approach Euclidean-margin optimality. Our main contribution is the quantitative description of Adam's behavior for small fixed positive ε. We give sufficient conditions under which an Adam-trained classifier nearly maximizes the max-norm margin before the updates become gradient-like. We also show that the classifier reaches a fixed target Euclidean margin only much later. Specifically, we show that for polynomially decreasing stepsizes with exponent (a), where (1/3<a<1), the updates become approximately proportional to the negative gradient after Theta(log(1/ε)1/(1-a)) iterations. At that time, the classifier still nearly maximizes the max-norm margin. Reaching a fixed target Euclidean margin above that of every max-norm-optimal classifier, but below the optimum, is shown to require ε-Theta(1)/(1-a) iterations. Under inverse-linear stepsize decay ((a=1)), the update transition takes polynomially many iterations, whereas reaching the target margin takes exponentially many. Experiments support these predictions. The later change in the classifier can improve or worsen generalization after training error reaches zero, connecting the analysis to grokking and its reverse.