# Beyond Dense Adam States: Adaptive Log-Space Quantization for Memory-Efficient Optimizers

arXiv: https://arxiv.org/abs/2608.22322 · HTML: https://mlsanity.com/paper/2608.22322/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 16 references that are in the map.
- First posted on arXiv 23 Aug 2026.

## Builds on (most cited first, 12 of 12)

- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection](https://mlsanity.com/paper/2403.03507/) (2024)
- [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021)
- [Memory Efficient Optimizers with 4-bit States](https://mlsanity.com/paper/2309.01507/) (2023)
- [APOLLO: SGD-like Memory, AdamW-level Performance](https://mlsanity.com/paper/2412.05270/) (2024)
- [CAME: Confidence-guided Adaptive Memory Efficient Optimization](https://mlsanity.com/paper/2307.02047/) (2023)
- [COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training](https://mlsanity.com/paper/2410.19313/) (2024)
- [Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets](https://mlsanity.com/paper/2603.16731/) (2026)
- [Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamics](https://mlsanity.com/paper/2505.00347/) (2025)
- [STQuant: Spatio-Temporal Adaptive Framework for Optimizer Quantization in Large Multimodal Model Training](https://mlsanity.com/paper/2604.06836/) (2026)

## Abstract (arXiv)

Optimizer-state quantization is commonly designed for Adam's dense, parameter-aligned first- and second-moment arrays. This abstraction breaks for memory-efficient optimizers, whose states may be factored, confidence-modulated, or maintained in a projected space, so similar reconstruction error can produce different update error. We formulate optimizer-state quantization as a joint problem over representation, topology, and update semantics. We then introduce Adaptive Log-Space (AL) quantization for non-negative states. AL fits each block's observed nonzero logarithmic interval and reserves a separate code for exact zero, enforcing q = 0 Leftrightarrow x = 0; signed momentum and state precision remain independently selectable. Controlled probes show that adaptive ranges reduce update error and temporal drift, exact-zero reservation preserves dormant states, and state topology constrains useful block granularity. End-to-end language-model training evaluates the resulting policy across dense, factored, confidence, and projected optimizer states. On TinyLlama-1.1B, AL8 with uniform 8-bit momentum reaches 72.90 perplexity versus 73.54 for bitsandbytes 8-bit AdamW, with comparable optimizer-state storage and higher throughput. CAME matches reference-level final perplexity across three seeds when its non-negative states use AL16, while a semantic grouping-and-protection policy closes most of quantized Adafactor's 100K-step late-loss gap. These results make state topology and update semantics first-class design constraints for optimizer quantization.