# M+Adam: Low-Precision Training via Additive-Multiplicative Optimization

arXiv: https://arxiv.org/abs/2607.10611 · HTML: https://mlsanity.com/paper/2607.10611/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 12 Jul 2026.

## Builds on (most cited first, 15 of 15)

- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection](https://mlsanity.com/paper/2403.03507/) (2024)
- [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021)
- [Benchmarking Optimizers for Large Language Model Pretraining](https://mlsanity.com/paper/2509.01440/) (2025)
- [Scaling FP8 training to trillion-token LLMs](https://mlsanity.com/paper/2409.12517/) (2024)
- [Memory Efficient Optimizers with 4-bit States](https://mlsanity.com/paper/2309.01507/) (2023)
- [APOLLO: SGD-like Memory, AdamW-level Performance](https://mlsanity.com/paper/2412.05270/) (2024)
- [Quartet: Native FP4 Training Can Be Optimal for Large Language Models](https://mlsanity.com/paper/2505.14669/) (2025)
- [Learning compositional functions via multiplicative weight updates](https://mlsanity.com/paper/2006.14560/) (2020)
- [LoQT: Low-Rank Adapters for Quantized Pretraining](https://mlsanity.com/paper/2405.16528/) (2024)
- [Stochastic Rounding for LLM Training: Theory and Practice](https://mlsanity.com/paper/2502.20566/) (2025)
- [LNS-Madam: Low-Precision Training in Logarithmic Number System using Multiplicative Weight Update](https://mlsanity.com/paper/2106.13914/) (2021)
- [TensorGRaD: Tensor Gradient Robust Decomposition for Memory-Efficient Neural Operator Training](https://mlsanity.com/paper/2501.02379/) (2025)
- [Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networks](https://mlsanity.com/paper/2506.17768/) (2025)
- [Multiplicative update rules for accelerating deep learning training and increasing robustness](https://mlsanity.com/paper/2307.07189/) (2023)

## Built on by (most cited first, 1 of 1)

- [Backpropagated Output Momentum: Relocating Optimizer History from Parameters to Task Space](https://mlsanity.com/paper/2609.36738/) (2026)

## Abstract (arXiv)

Training with quantized weights can reduce costs but often results in degraded accuracy, especially when optimization is carried out in low precision, without storing high-precision copies. We identify a key failure mode: under low precision, standard optimizers can get stuck and not make progress, especially at large weight magnitudes due to coarse mantissa resolution. To overcome this, multiplicative updates have been previously proposed, in place of additive updates in standard optimizers. While successful under extremely low precision, such as under the logarithmic number system, they suffer from failures near zero and across sign changes. The failure modes of additive and multiplicative updates are therefore complementary. To exploit this, we propose M+Adam, which combines both update types: additive steps handle sign changes and small magnitudes, while multiplicative steps ensure progress at large magnitudes when additive updates are zeroed out under rounding. We prove monotone descent for M+Adam under standard smoothness assumptions. Across LLaMA-style pretraining with 60M-1B models, 1x-8x Chinchilla budgets, and using only BF16, FP8, and FP4 master weights, M+Adam consistently improves low-precision training.