mlsanityData to 7 Oct 2026

The Implicit Bias of Adam on Separable Data

arXiv 2406.10650 ↗First posted 15 Jun 2024stat.ML cs.LGcited by 32

Not on the map. Why?

Abstract (arXiv)

Adam has become one of the most favored optimizers in deep learning problems. Despite its success in practice, numerous mysteries persist regarding its theoretical understanding. In this paper, we study the implicit bias of Adam in linear logistic regression. Specifically, we show that when the training data are linearly separable, Adam converges towards a linear classifier that achieves the maximum ℓ_∞-margin. Notably, for a general class of diminishing learning rates, this convergence occurs within polynomial time. Our result shed light on the difference between Adam and (stochastic) gradient descent from a theoretical perspective.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 47 of its 57 references that are in the map. First posted on arXiv 15 Jun 2024.

Builds on

47 of 57 references.

References · 47

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019On the Convergence of Adam and Beyondcited by 834
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  4. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  5. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  6. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  7. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  8. 2017Implicit Regularization in Matrix Factorizationcited by 318
  9. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  10. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  11. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  12. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
35 more
  1. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  2. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  3. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 173
  4. 2020Directional convergence and alignment in deep learningcited by 172
  5. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 136
  6. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 125
  7. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  8. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  9. 2022Adam Can Converge Without Any Modification On Update Rulescited by 112
  10. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
  11. 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 63
  12. 2020The Implicit Regularization of Stochastic Gradient Flow for Least Squarescited by 58
  13. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
  14. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
  15. 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 50
  16. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 50
  17. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 48
  18. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  19. 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 43
  20. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  21. 2021SUPER-ADAM: Faster and Universal Framework of Adaptive Gradientscited by 40
  22. 2018Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov accelerationcited by 39
  23. 2020The Geometry of Sign Gradient Descentcited by 39
  24. 2013Margins, Shrinkage, and Boostingcited by 35
  25. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  26. 2022Gradient Methods Provably Converge to Non-Robust Networkscited by 27
  27. 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 21
  28. 2024On Convergence of Adam for Stochastic Optimization under Relaxed Assumptionscited by 21
  29. 2019Implicit Regularization and Convergence for Weight Normalizationcited by 20
  30. 2023Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Datacited by 18
  31. 2020Adam^+: A Stochastic Method with Adaptive Variance Reductioncited by 17
  32. 2019The Implicit Bias of AdaGrad on Separable Datacited by 15
  33. 2020Implicit Bias of Gradient Descent for Mean Squared Error Regression with Two-Layer Wide Neural Networkscited by 15
  34. 2023The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networkscited by 8
  35. 2023Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescalingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 12; 2026: 15.

32 papers.

Sort:

By area:

  1. 2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 34
  2. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 33
  3. 2025How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Datacited by 16
  4. 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 13
  5. 2025The Rich and the Simple: On the Implicit Bias of Adam and SGDcited by 12
  6. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  7. 2026The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networkscited by 6
  8. 2026Can Muon Fine-tune Adam-Pretrained Models?cited by 6
  9. 2025A Tale of Two Geometries: Adaptive Optimizers and Non-Euclidean Descentcited by 5
  10. 2025Towards Understanding Generalization in DP-GD: A Case Study in Training Two-Layer CNNscited by 4
  11. 2026OLion: Approaching the Hadamard Ideal by Intersecting Spectral and ℓ∞ Implicit Biasescited by 3
  12. 2024Continuous-Time Analysis of Adaptive Optimization and Normalizationcited by 2
20 more
  1. 2025Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regimecited by 2
  2. 2025Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamicscited by 2
  3. 2026Never Saddle for Reparameterized Steepest Descent as Mirror Flowcited by 2
  4. 2024Effectiveness of Distributed Gradient Descent with Local Steps for Overparameterized Modelscited by 1
  5. 2024Towards Simple and Provable Parameter-Free Adaptive Gradient Methodscited by 1
  6. 2025How do simple rotations affect the implicit bias of Adam?cited by 1
  7. 2025Gradient Descent Robustly Learns the Intrinsic Dimension of Data in Training Convolutional Neural Networkscited by 0
  8. 2025Heavy-Ball Momentum Method in Continuous Time and Discretization Error Analysiscited by 0
  9. 2025When Are Learning Biases Equivalent? A Unifying Framework for Fairness, Robustness, and Distribution Shiftcited by 0
  10. 2026Dual Space Preconditioning for Gradient Descent in the Overparameterized Regimecited by 0
  11. 2026Towards The Implicit Bias on Multiclass Separable Data Under Norm Constraintscited by 0
  12. 2026Understanding Dynamics of Adam in Zero-Sum Games: An ODE Approachcited by 0
  13. 2026Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learningcited by 0
  14. 2026HORST: Composing Optimizer Geometries for Sparse Transformer Trainingcited by 0
  15. 2026The Implicit Bias of Depth: From Neural Collapse to Softmax Codescited by 0
  16. 2026The Anatomy of Implicit Bias: Information Allocation in Neural Network Trainingcited by 0
  17. 2026Non-asymptotic implicit bias of logistic regression at early-stage gradient descent dynamicscited by 0
  18. 2026The Loss Does Not See the Basis, but Adam Doescited by 0
  19. 2026Benign Overfitting for General Norms and Distributionscited by 0
  20. 2026Optimizer-dependent training dynamics converge to the same one-third optimal data scalingcited by 0