The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networks

arXiv 2602.16340 ↗First posted 18 Feb 2026cs.LG stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

We study the implicit bias of momentum-based optimizers on smooth homogeneous models. We show that textitmomentum steepest descent algorithms like Muon (spectral norm), MomentumGD (ℓ2 norm), and Signum (ℓ_∞ norm) are textitapproximate steepest descent trajectories under a decaying learning rate schedule, proving that these algorithms have a bias towards KKT points of the corresponding margin maximization problem. We extend the analysis to Adam (without the stability constant), which maximizes the ℓ_∞ margin, and to Muon-Signum and Muon-Adam, which maximize a hybrid norm. Our experiments corroborate the theory and show that the identity of the margin maximized depends on the choice of optimizer. Overall, our results extend earlier lines of work on steepest descent in homogeneous models and momentum-based optimizers in linear models.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 54 of its 65 references that are in the map. First posted on arXiv 18 Feb 2026.

Builds on

54 of 65 references.

References · 54

2014Adam: A Method for Stochastic Optimizationcited by 36,960
2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
2019On the Convergence of Adam and Beyondcited by 834
2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
2025Muon is Scalable for LLM Trainingcited by 381
2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248

38 more

2018Gradient descent aligns the layers of deep linear networkscited by 182
2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
2024Old Optimizer, New Norm: An Anthologycited by 173
2020Directional convergence and alignment in deep learningcited by 172
2020A Simple Convergence Proof of Adam and Adagradcited by 169
2022Reconstructing Training Data from Trained Neural Networkscited by 129
2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 125
2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 124
2018Stochastic subgradient method converges on tame functionscited by 120
2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 117
2022Adam Can Converge Without Any Modification On Update Rulescited by 112
2018Convergence of Gradient Descent on Separable Datacited by 108
2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 74
2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 70
2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
2020Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functionscited by 50
2018Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimizationcited by 47
2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 43
2025In Search of Adam's Secret Saucecited by 40
2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 34
2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 33
2024The Implicit Bias of Adam on Separable Datacited by 32
2022Gradient Methods Provably Converge to Non-Robust Networkscited by 27
2022The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networkscited by 25
2023Deconstructing Data Reconstruction: Multiclass, Weight Decay and General Lossescited by 24
2023The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networkscited by 23
2025Learning quadratic neural networks in high dimensions: SGD dynamics and scaling lawscited by 23
2023On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensionscited by 21
2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 13
2019Stationary Points of Shallow Neural Networks with Quadratic Activation Functioncited by 12
2024Reconstructing Training Data From Real World Models Trained with Transfer Learningcited by 10
2022Non-Convergence and Limit Cycles in the Adam optimizercited by 9
2025Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networkscited by 9

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

2026Scale-Invariant Neural Network Optimization: Norm Geometry and Heavy-Tailed Noisecited by 2
2026The Spectral Dynamics and Noise Geometry of Muoncited by 2
2026Convergence of Spectral Descent for Non-smooth Optimizationcited by 1
2026The Loss Does Not See the Basis, but Adam Doescited by 0
2026Is Grokking a Loss of Normal Hyperbolicity of the Interpolation Manifold?cited by 0
2026Benign Overfitting for General Norms and Distributionscited by 0