# A Convergence Analysis of Adaptive Optimizers under Floating-point Quantization

arXiv: https://arxiv.org/abs/2510.21314 · HTML: https://mlsanity.com/paper/2510.21314/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2025.

## Built on by (most cited first, 8 of 8)

- [Improved Convergence Rates of Muon Optimizer for Nonconvex Optimization](https://mlsanity.com/paper/2601.19400/) (2026)
- [Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets](https://mlsanity.com/paper/2603.16731/) (2026)
- [StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models](https://mlsanity.com/paper/2604.15416/) (2026)
- [Muon Converges under Heavy-Tailed Noise: Nonconvex H"older-Smooth Empirical Risk Minimization](https://mlsanity.com/paper/2603.15059/) (2026)
- [MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization](https://mlsanity.com/paper/2605.11396/) (2026)
- [Information-Theoretic Lower Bounds for Bit-Constrained Stochastic Optimization via a Reduction to Compressed Gaussian Mean Estimation](https://mlsanity.com/paper/2606.00703/) (2026)
- [FedAdaVR: Adaptive Variance Reduction for Robust Federated Learning under Limited Client Participation](https://mlsanity.com/paper/2601.22204/) (2026)
- [PowerStep: Memory-Efficient Adaptive Optimization via ℓp-Norm Steepest Descent](https://mlsanity.com/paper/2605.10335/) (2026)

## Abstract (arXiv)

The rapid scaling of large language models (LLMs) has made low-precision training essential for reducing memory, improving efficiency, and enabling larger models and datasets. Existing convergence theories for adaptive optimizers, however, assume all components are exact and neglect hardware-aware quantization, leaving open the question of why low-precision training remains effective. We introduce the first theoretical framework for analyzing the convergence of adaptive optimizers, including Adam and Muon, under floating-point quantization of gradients, weights, and optimizer states (e.g., moment estimates). Within this framework, we derive convergence rates on smooth non-convex objectives under standard stochastic gradient assumptions, explicitly characterizing how quantization errors from different components affect convergence. We show that both algorithms retain rates close to their full-precision counterparts provided mantissa length scales only logarithmically with the number of iterations. Our analysis further reveals that Adam is highly sensitive to weights and second-moment quantization due to its reliance on β2 → 1, while Muon requires weaker error control and is thus potentially more robust. These results narrow the gap between empirical success and theoretical understanding of low-precision training methods. Numerical experiments on synthetic and real-world data corroborate our theory.