mlsanityData to 14 Oct 2026

Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimization

arXiv 2404.04454 ↗First posted 5 Apr 2024cs.LG math.OC stat.MLcited by 58

Not on the map. Why?

Abstract (arXiv)

Adam with decoupled weight decay, also known as AdamW, is widely acclaimed for its superior performance in language modeling tasks, surpassing Adam with ℓ2 regularization in terms of generalization and optimization. However, this advantage is not theoretically well-understood. One challenge here is that though intuitively Adam with ℓ2 regularization optimizes the ℓ2 regularized loss, it is not clear if AdamW optimizes a specific objective. In this work, we make progress toward understanding the benefit of AdamW by showing that it implicitly performs constrained optimization. More concretely, we show in the full-batch setting, if AdamW converges with any non-increasing learning rate schedule whose partial sum diverges, it must converge to a KKT point of the original loss under the constraint that the ℓ_∞ norm of the parameter is bounded by the inverse of the weight decay factor. This result is built on the observation that Adam can be viewed as a smoothed version of SignGD, which is the normalized steepest descent with respect to ℓ_∞ norm, and a surprising connection between normalized steepest descent with weight decay and Frank-Wolfe.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Builds on 49 of its 67 references that are in the map. First posted on arXiv 5 Apr 2024.

Builds on

49 of 67 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2016Layer Normalizationcited by 5,669
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  5. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  6. 2019On the Convergence of Adam and Beyondcited by 841
  7. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  8. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  9. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  10. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  11. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  12. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
37 more
  1. 2017Implicit Regularization in Matrix Factorizationcited by 322
  2. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  3. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  4. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  5. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
  6. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 202
  7. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  8. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  9. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
  10. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  11. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  12. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  13. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 139
  14. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  15. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 126
  16. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  17. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  18. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  19. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  20. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  21. 2018Three Mechanisms of Weight Decay Regularizationcited by 113
  22. 2018Convergence of Gradient Descent on Separable Datacited by 110
  23. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  24. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
  25. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 83
  26. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 81
  27. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  28. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  29. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  30. 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
  31. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 51
  32. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
  33. 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 46
  34. 2022Understanding AdamW through Proximal Methods and Scale-Freenesscited by 46
  35. 2022Robust Training of Neural Networks Using Scale Invariant Architecturescited by 29
  36. 2023On the Implicit Bias of Adamcited by 27
  37. 2022The Iterates of the Frank-Wolfe Algorithm May Not Convergecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 21; 2026: 26.

58 papers.

Sort:

By area:

  1. 2024Old Optimizer, New Norm: An Anthologycited by 193
  2. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  3. 2025Muon Optimizes Under Spectral Norm Constraintscited by 86
  4. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 41
  5. 2025In Search of Adam's Secret Saucecited by 40
  6. 2024Does SGD really happen in tiny subspaces?cited by 36
  7. 2024The Implicit Bias of Adam on Separable Datacited by 36
  8. 2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 36
  9. 2024Adam Exploits ℓ_∞-geometry of Loss Landscape via Coordinate-wise Adaptivitycited by 23
  10. 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 18
  11. 2025Generalized Gradient Norm Clipping & Non-Euclidean (L0,L1)-Smoothnesscited by 17
  12. 2025Is your batch size the problem? Revisiting the Adam-SGD gap in language modelingcited by 15
38 more
  1. 2025The Rich and the Simple: On the Implicit Bias of Adam and SGDcited by 13
  2. 2025Cautious Weight Decaycited by 13
  3. 2024Convergence Rate Analysis of LIONcited by 11
  4. 2025Generalization of Diffusion Models Arises with a Balanced Representation Spacecited by 11
  5. 2026The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networkscited by 11
  6. 2025Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learningcited by 9
  7. 2025On the O(frac√dK1/4) Convergence Rate of AdamW Measured by ℓ1 Normcited by 8
  8. 2025FedMuon: Federated Learning with Bias-corrected LMO-based Optimizationcited by 8
  9. 2025How Memory in Optimization Algorithms Implicitly Modifies the Losscited by 6
  10. 2025On Learning Verifiers and Implications to Chain-of-Thought Reasoningcited by 6
  11. 2025A Tale of Two Geometries: Adaptive Optimizers and Non-Euclidean Descentcited by 5
  12. 2025Provable Benefit of SignGD: A Minimal Model Under Heavy-Tailed Class Imbalancecited by 5
  13. 2024Continuous-Time Analysis of Adaptive Optimization and Normalizationcited by 4
  14. 2025Continuous Representation Methods, Theories, and Applications: An Overview and Perspectivescited by 4
  15. 2025Towards Understanding Generalization in DP-GD: A Case Study in Training Two-Layer CNNscited by 4
  16. 2026The Effect of Mini-Batch Noise on the Implicit Bias of Adamcited by 4
  17. 2026Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikescited by 4
  18. 2024A Mirror Descent Perspective of Smoothed Sign Descentcited by 3
  19. 2026OLion: Approaching the Hadamard Ideal by Intersecting Spectral and ℓ∞ Implicit Biasescited by 3
  20. 2026Optimistic Dual Averaging Unifies Modern Optimizerscited by 3
  21. 2026Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learningcited by 3
  22. 2026Muon Learns More Robust and Transferable Features than Adamcited by 3
  23. 2025Rethinking Benign Overfitting in Two-Layer Neural Networkscited by 2
  24. 2025Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regimecited by 2
  25. 2026Tight Complexity of Muon under Heavy-Tailed Noisecited by 2
  26. 2024LoKO: Low-Rank Kalman Optimizer for Online Fine-Tuning of Large Modelscited by 1
  27. 2024Effectiveness of Distributed Gradient Descent with Local Steps for Overparameterized Modelscited by 1
  28. 2026Convergence Rate Analysis of the AdamW-style Shampoo: Unifying One-Sided and Two-Sided Preconditioningcited by 1
  29. 2026DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Modelscited by 1
  30. 2026HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalizationcited by 1
  31. 2026Demystifying Manifold Constraints in LLM Pre-trainingcited by 1
  32. 2026First-Passage Prediction of Grokking Delay: ACalibrated Law under AdamW with Causal Validationcited by 1
  33. 2026Finite-Horizon Input-Output Dynamics of Minibatch Perturbations in AdamWcited by 1
  34. 2025Gradient Descent Robustly Learns the Intrinsic Dimension of Data in Training Convolutional Neural Networkscited by 0
  35. 2026Dual Space Preconditioning for Gradient Descent in the Overparameterized Regimecited by 0
  36. 2026Accelerating LMO-Based Optimization via Implicit Gradient Transportcited by 0
  37. 2026Gradient Extrapolation-Based Policy Optimizationcited by 0
  38. 2026Optimizer-Induced Mode Connectivity: From AdamW to Muoncited by 0

50 most cited of 58 All 58 (JSON)