mlsanityData to 14 Oct 2026

Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets

arXiv 2603.16731 ↗First posted 17 Mar 2026cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Quantizing optimizer states is becoming an important ingredient of memory-efficient large-scale pre-training, but the resulting optimizer dynamics remain only partially understood. We study low-precision exponential moving average (EMA) optimizer states and show how quantization can cause many nominal updates to round back to the same stored value, making the state effectively stale and slowing adaptation beyond what the nominal decay would suggest. We then develop a simple predictive model of stalling that estimates one-step stalling probabilities and characterizes how stalling builds up over time after the initialization. This perspective provides a mechanistic explanation for why optimizer-state resets help in low precision: once a quantized EMA becomes effectively stale, resetting it can temporarily restore responsiveness. Motivated by this picture, we derive a simple theory-guided method for choosing useful reset periods, showing that in low precision the key question is not only whether resets help, but when they should be applied. Experiments in controlled simulations and LLM pre-training show that suitable reset schedules recover the performance lost to low-precision state storage while substantially reducing optimizer-state memory.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 30 of its 47 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

30 of 47 references.

References · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  6. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  7. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  8. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  9. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  10. 2015Deep Learning with Limited Numerical Precisioncited by 376
  11. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  12. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
18 more
  1. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  2. 2024Scaling FP8 training to trillion-token LLMscited by 65
  3. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 56
  4. 2023Memory Efficient Optimizers with 4-bit Statescited by 54
  5. 2025FP4 All the Way: Fully Quantized Training of LLMscited by 40
  6. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 39
  7. 2024DB-LLM: Accurate Dual-Binarization for Efficient LLMscited by 37
  8. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 33
  9. 2025SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Trainingcited by 28
  10. 2023Resetting the Optimizer in Deep RL: An Empirical Studycited by 26
  11. 20244-bit Shampoo for Memory-Efficient Network Trainingcited by 18
  12. 2022On the Overlooked Structure of Stochastic Gradientscited by 13
  13. 2025Effective Quantization of Muon Optimizer Statescited by 11
  14. 2024Collage: Light-Weight Low-Precision Strategy for LLM Trainingcited by 10
  15. 2025A Convergence Analysis of Adaptive Optimizers under Floating-point Quantizationcited by 8
  16. 2025Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamicscited by 5
  17. 2025Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarkingcited by 5
  18. 2025Adaptive Memory Momentum via a Model-Based Framework for Deep Learning Optimizationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026PowerStep: Memory-Efficient Adaptive Optimization via ℓp-Norm Steepest Descentcited by 0
  2. 2026Outer-Momentum Restarting in High-Dimensional Two-Phase Optimizationcited by 0
  3. 2026Auditing Invisible Weight Updates with Reference Tracescited by 0
  4. 2026Beyond Dense Adam States: Adaptive Log-Space Quantization for Memory-Efficient Optimizerscited by 0
  5. 2026Blog: Survey of Optimizerscited by 0
  6. 2026Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting
  2. 2025Beyond Outliers: A Study of Optimizers Under Quantization
  3. 2025When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models
  4. 2026Quantization-Robust Unlearning through the Lens of Retain-Forget Loss Landscapes Interaction
  5. 2026Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximation
  6. 2026Beyond Precision: Training-Inference Mismatch is an Optimization Problem and Simple LR Scheduling Fixes It
  7. 2026Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy
  8. 2026Quantized Evolution Strategies: High-precision Fine-tuning of Quantized LLMs at Low-precision Cost