mlsanityData to 14 Oct 2026

How Memory in Optimization Algorithms Implicitly Modifies the Loss

arXiv 2502.02132 ↗First posted 4 Feb 2025cs.LG cs.AI math.OCcited by 6

Not on the map. Why?

Abstract (arXiv)

In modern optimization methods used in deep learning, each update depends on the history of previous iterations, often referred to as memory, and this dependence decays fast as the iterates go further into the past. For example, gradient descent with momentum has exponentially decaying memory through exponentially averaged past gradients. We introduce a general technique for identifying a memoryless algorithm that approximates an optimization algorithm with memory. It is obtained by replacing all past iterates in the update by the current one, and then adding a correction term arising from memory (also a function of the current iterate). This correction term can be interpreted as a perturbation of the loss, and the nature of this perturbation can inform how memory implicitly (anti-)regularizes the optimization dynamics. As an application of our theory, we find that Lion does not have the kind of implicit anti-regularization induced by memory that AdamW does, providing a theory-based explanation for Lion's better generalization performance recently documented.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 51 of its 70 references that are in the map. First posted on arXiv 4 Feb 2025.

Builds on

51 of 70 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  7. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  10. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  11. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  12. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
38 more
  1. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  2. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  3. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  4. 2017Implicit Regularization in Matrix Factorizationcited by 322
  5. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  6. 2020Understanding the Difficulty of Training Transformerscited by 210
  7. 2024Old Optimizer, New Norm: An Anthologycited by 193
  8. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  9. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  10. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  11. 2020Implicit Gradient Regularizationcited by 146
  12. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  13. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  14. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  15. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  16. 2018Convergence of Gradient Descent on Separable Datacited by 110
  17. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  18. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
  19. 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 103
  20. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 101
  21. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
  22. 2018Risk and parameter convergence of logistic regressioncited by 85
  23. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 77
  24. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 65
  25. 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 64
  26. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
  27. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 51
  28. 2018Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimizationcited by 49
  29. 2020A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithmscited by 33
  30. 2019Continuous Time Analysis of Momentum Methodscited by 32
  31. 2023On the Implicit Bias of Adamcited by 27
  32. 2023Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descentcited by 25
  33. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 23
  34. 2024Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noisecited by 23
  35. 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 22
  36. 2019The Implicit Bias of AdaGrad on Separable Datacited by 16
  37. 2022Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysiscited by 13
  38. 2023On a continuous time model of gradient descent dynamics and instability in deep learningcited by 11

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectorycited by 8
  2. 2026Optimizer Memory Makes Shuffle Order a First-Order Source of Fine-Tuning Noisecited by 7
  3. 2026The Effect of Mini-Batch Noise on the Implicit Bias of Adamcited by 4
  4. 2026Stored in Optimizer State, Valued by Later Training: A Causal Account of Subliminal Trait Transfercited by 3
  5. 2026Same Error, Different Function: The Optimizer as an Implicit Prior in Financial Time Seriescited by 0
  6. 2026Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Biascited by 0