Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regime
Not on the map. Why?
Abstract (arXiv)
Adam [Kingma & Ba, 2015] is the de facto optimizer in deep learning, yet its theoretical understanding remains limited. Prior analyses show that Adam favors solutions aligned with ℓ_∞-geometry, but these results are restricted to the full-batch regime. In this work, we study the implicit bias of incremental Adam (using one sample per step) for logistic regression on linearly separable data, and show that its bias can deviate from the full-batch behavior. As an extreme example, we construct datasets on which incremental Adam provably converges to the ℓ2-max-margin classifier, in contrast to the ℓ_∞-max-margin bias of full-batch Adam. For general datasets, we characterize its bias using a proxy algorithm for the β2 → 1 limit. This proxy maximizes a data-adaptive Mahalanobis-norm margin, whose associated covariance matrix is determined by a data-dependent dual fixed-point formulation. We further present concrete datasets where this bias reduces to the standard ℓ2- and ℓ_∞-max-margin classifiers. As a counterpoint, we prove that Signum [Bernstein et al., 2018] converges to the ℓ_∞-max-margin classifier for any batch size. Overall, our results highlight that the implicit bias of Adam crucially depends on both the batching scheme and the dataset, while Signum remains invariant.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 42 of its 49 references that are in the map. First posted on arXiv 30 Oct 2025.
Builds on
42 of 49 references.
References · 42
- 2016Understanding deep learning requires rethinking generalizationcited by 1,674
- 2019On the Convergence of Adam and Beyondcited by 841
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
- 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 641
- 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
- 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
- 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
- 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
- 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
- 2018Gradient descent aligns the layers of deep linear networkscited by 182
- 2020Directional convergence and alignment in deep learningcited by 178
- 2020A Simple Convergence Proof of Adam and Adagradcited by 171
30 more
- 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 139
- 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
- 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
- 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
- 2023Convergence of Adam Under Relaxed Assumptionscited by 99
- 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 98
- 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
- 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 80
- 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 75
- 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 64
- 2020Gradient descent follows the regularization path for general lossescited by 59
- 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
- 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 53
- 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 49
- 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 46
- 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 41
- 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 41
- 2025In Search of Adam's Secret Saucecited by 40
- 2021Understanding the role of importance weighting for deep learningcited by 39
- 2024The Implicit Bias of Adam on Separable Datacited by 36
- 2022Understanding Why Generalized Reweighting Does Not Improve Over ERMcited by 29
- 2022How Does Adaptive Optimization Impact Local Neural Network Geometry?cited by 24
- 2024Adam with model exponential moving average is effective for nonconvex optimizationcited by 23
- 2024Adam Exploits ℓ_∞-geometry of Loss Landscape via Coordinate-wise Adaptivitycited by 23
- 2024On Convergence of Adam for Stochastic Optimization under Relaxed Assumptionscited by 21
- 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 18
- 2025Is your batch size the problem? Revisiting the Adam-SGD gap in language modelingcited by 15
- 2025The Rich and the Simple: On the Implicit Bias of Adam and SGDcited by 13
- 2024Understanding Adam Requires Better Rotation Dependent Assumptionscited by 11
- 2024A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGDcited by 9
Built on by
Citing papers per year: 2026: 2.
2 papers.