# LaProp: Separating Momentum and Adaptivity in Adam

arXiv: https://arxiv.org/abs/2002.04839 · HTML: https://mlsanity.com/paper/2002.04839/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2020.

## Built on by (most cited first, 20 of 20)

- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Robustness to Unbounded Smoothness of Generalized SignSGD](https://mlsanity.com/paper/2208.11195/) (2022)
- [Scaling Behavior of Discrete Diffusion Language Models](https://mlsanity.com/paper/2512.10858/) (2025)
- [Random Scaling and Momentum for Non-smooth Non-convex Optimization](https://mlsanity.com/paper/2405.09742/) (2024)
- [FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher information](https://mlsanity.com/paper/2405.12807/) (2024)
- [Deriving Hyperparameter Scaling Laws via Modern Optimization Theory](https://mlsanity.com/paper/2603.15958/) (2026)
- [SNOO: Step-K Nesterov Outer Optimizer - The Surprising Effectiveness of Nesterov Momentum Applied to Pseudo-Gradients](https://mlsanity.com/paper/2510.15830/) (2025)
- [Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectory](https://mlsanity.com/paper/2602.09314/) (2026)
- [On Surprising Effectiveness of Masking Updates in Adaptive Optimizers](https://mlsanity.com/paper/2602.15322/) (2026)
- [SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales](https://mlsanity.com/paper/2607.20548/) (2026)
- [Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads](https://mlsanity.com/paper/2405.20053/) (2024)
- [What makes a good feedforward computational graph?](https://mlsanity.com/paper/2502.06751/) (2025)
- [Smoothing DiLoCo with Primal Averaging for Faster Training of LLMs](https://mlsanity.com/paper/2512.17131/) (2025)
- [Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations](https://mlsanity.com/paper/2604.12968/) (2026)
- [Torque-Aware Momentum](https://mlsanity.com/paper/2412.18790/) (2024)
- [Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss](https://mlsanity.com/paper/2606.06418/) (2026)
- [Optimizer Memory Schedules for Outscaling the Overtraining Axis](https://mlsanity.com/paper/2609.04577/) (2026)
- [NashDreamer: Model-Based Reinforcement Learning for Zero-Sum Imperfect-Information Games](https://mlsanity.com/paper/2609.01549/) (2026)
- [The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics](https://mlsanity.com/paper/2609.35392/) (2026)
- [How to scale your HEP ML models: A recipe for robust architecture comparisons at scale](https://mlsanity.com/paper/2610.06784/) (2026)

## Abstract (arXiv)

We identity a by-far-unrecognized problem of Adam-style optimizers which results from unnecessary coupling between momentum and adaptivity. The coupling leads to instability and divergence when the momentum and adaptivity parameters are mismatched. In this work, we propose a method, Laprop, which decouples momentum and adaptivity in the Adam-style methods. We show that the decoupling leads to greater flexibility in the hyperparameters and allows for a straightforward interpolation between the signed gradient methods and the adaptive gradient methods. We experimentally show that Laprop has consistently improved speed and stability over Adam on a variety of tasks. We also bound the regret of Laprop on a convex problem and show that our bound differs from that of Adam by a key factor, which demonstrates its advantage.