mlsanityData to 14 Oct 2026

On the Implicit Bias of Adam

arXiv 2309.00079 ↗First posted 31 Aug 2023cs.LG cs.AI math.OCcited by 27

Not on the map. Why?

Abstract (arXiv)

In previous literature, backward error analysis was used to find ordinary differential equations (ODEs) approximating the gradient descent trajectory. It was found that finite step sizes implicitly regularize solutions because terms appearing in the ODEs penalize the two-norm of the loss gradients. We prove that the existence of similar implicit regularization in RMSProp and Adam depends on their hyperparameters and the training stage, but with a different "norm" involved: the corresponding ODE terms either penalize the (perturbed) one-norm of the loss gradients or, conversely, impede its reduction (the latter case being typical). We also conduct numerical experiments and discuss how the proven facts can influence generalization.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 52 of its 63 references that are in the map. First posted on arXiv 31 Aug 2023.

Builds on

52 of 63 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  6. 2019On the Convergence of Adam and Beyondcited by 841
  7. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  8. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  9. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  10. 2014Deeply-Supervised Netscited by 367
  11. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  12. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
38 more
  1. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  2. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  3. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  4. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  5. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
  6. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  7. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  8. 2020Directional convergence and alignment in deep learningcited by 178
  9. 2020Implicit Gradient Regularizationcited by 146
  10. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  11. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 126
  12. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 125
  13. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  14. 2018Convergence of Gradient Descent on Separable Datacited by 110
  15. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
  16. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 101
  17. 2022Better plain ViT baselines for ImageNet-1kcited by 98
  18. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 96
  19. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
  20. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 90
  21. 2018Risk and parameter convergence of logistic regressioncited by 85
  22. 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 64
  23. 2023A Theory on Adam Instability in Large-Scale Machine Learningcited by 56
  24. 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
  25. 2018Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimizationcited by 49
  26. 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 46
  27. 2020The Geometry of Sign Gradient Descentcited by 41
  28. 2020Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentumcited by 40
  29. 2020A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithmscited by 33
  30. 2019Continuous Time Analysis of Momentum Methodscited by 32
  31. 2023Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descentcited by 25
  32. 2022How Does Adaptive Optimization Impact Local Neural Network Geometry?cited by 24
  33. 2020On dissipative symplectic integration with applications to gradient-based optimizationcited by 23
  34. 2022How to Fine-Tune Vision Models with SGDcited by 23
  35. 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 22
  36. 2020Flatness is a False Friendcited by 19
  37. 2019The Implicit Bias of AdaGrad on Separable Datacited by 16
  38. 2021Discretization Drift in Two-Player Gamescited by 14

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 5; 2026: 12.

27 papers.

Sort:

  1. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
  2. 2024Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representationscited by 24
  3. 2024Implicit Optimization Bias of Next-Token Prediction in Linear Modelscited by 14
  4. 2025Stochastic Rounding for LLM Training: Theory and Practicecited by 10
  5. 2026Optimizer Memory Makes Shuffle Order a First-Order Source of Fine-Tuning Noisecited by 7
  6. 2025How Memory in Optimization Algorithms Implicitly Modifies the Losscited by 6
  7. 2025Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEscited by 5
  8. 2023Implicit biases in multitask and continual learning from a backward error analysis perspectivecited by 4
  9. 2024The Impact of Geometric Complexity on Neural Collapse in Transfer Learningcited by 4
  10. 2026The Effect of Mini-Batch Noise on the Implicit Bias of Adamcited by 4
  11. 2024How Do the Architecture and Optimizer Affect Representation Learning? On the Training Dynamics of Representations in Deep Neural Networkscited by 2
  12. 2026Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Datacited by 2
15 more
  1. 2024Effectiveness of Distributed Gradient Descent with Local Steps for Overparameterized Modelscited by 1
  2. 2025Learning by solving differential equationscited by 1
  3. 2026Implicit Regularization of Mini-Batch Training in Graph Neural Networkscited by 1
  4. 2024Majority Kernels: An Approach to Leverage Big Model Dynamics for Efficient Small Model Trainingcited by 0
  5. 2024Corridor Geometry in Gradient-Based Optimizationcited by 0
  6. 2024Error dynamics of mini-batch gradient descent with random reshuffling for least squares regressioncited by 0
  7. 2025Zero Generalization Error Theorem for Random Interpolators via Algebraic Geometrycited by 0
  8. 2026Same Error, Different Function: The Optimizer as an Implicit Prior in Financial Time Seriescited by 0
  9. 2026Optimizer-Induced Mode Connectivity: From AdamW to Muoncited by 0
  10. 2026Understanding Dynamics of Adam in Zero-Sum Games: An ODE Approachcited by 0
  11. 2026Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Biascited by 0
  12. 2026The Anatomy of Implicit Bias: Information Allocation in Neural Network Trainingcited by 0
  13. 2026The Loss Does Not See the Basis, but Adam Doescited by 0
  14. 2026Muon Learns Facts Better: Understanding the Role of Spectral Orthogonalizationcited by 0
  15. 2026Optimizer Geometry Sets the Pace: Spectral Learning Dynamics in Matrix Factorizationcited by 0