mlsanityData to 14 Oct 2026

LaProp: Separating Momentum and Adaptivity in Adam

arXiv 2002.04839 ↗First posted 12 Feb 2020cs.LG stat.MLcited by 20

Not on the map. Why?

Abstract (arXiv)

We identity a by-far-unrecognized problem of Adam-style optimizers which results from unnecessary coupling between momentum and adaptivity. The coupling leads to instability and divergence when the momentum and adaptivity parameters are mismatched. In this work, we propose a method, Laprop, which decouples momentum and adaptivity in the Adam-style methods. We show that the decoupling leads to greater flexibility in the hyperparameters and allows for a straightforward interpolation between the signed gradient methods and the adaptive gradient methods. We experimentally show that Laprop has consistently improved speed and stability over Adam on a variety of tasks. We also bound the regret of Laprop on a convex problem and show that our bound differs from that of Adam by a key factor, which demonstrates its advantage.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 4; 2025: 4; 2026: 10.

20 papers.

Sort:

By area:

  1. 2023Mastering Diverse Domains through World Modelscited by 1,253
  2. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 81
  3. 2025Scaling Behavior of Discrete Diffusion Language Modelscited by 35
  4. 2024Random Scaling and Momentum for Non-smooth Non-convex Optimizationcited by 19
  5. 2024FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher informationcited by 18
  6. 2026Deriving Hyperparameter Scaling Laws via Modern Optimization Theorycited by 10
  7. 2025SNOO: Step-K Nesterov Outer Optimizer - The Surprising Effectiveness of Nesterov Momentum Applied to Pseudo-Gradientscited by 8
  8. 2026Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectorycited by 8
  9. 2026On Surprising Effectiveness of Masking Updates in Adaptive Optimizerscited by 7
  10. 2026SOAP, Muon, and Beyond: Pushing LLM Pretraining Scalescited by 6
  11. 2024Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Headscited by 5
  12. 2025What makes a good feedforward computational graph?cited by 5
8 more
  1. 2025Smoothing DiLoCo with Primal Averaging for Faster Training of LLMscited by 5
  2. 2026Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluationscited by 3
  3. 2024Torque-Aware Momentumcited by 2
  4. 2026Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Losscited by 1
  5. 2026Optimizer Memory Schedules for Outscaling the Overtraining Axiscited by 1
  6. 2026NashDreamer: Model-Based Reinforcement Learning for Zero-Sum Imperfect-Information Gamescited by 0
  7. 2026The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamicscited by 0
  8. 2026How to scale your HEP ML models: A recipe for robust architecture comparisons at scalecited by 0