# Low-Bit Recurrent States in Hybrid Language Models

arXiv: https://arxiv.org/abs/2609.30950 · HTML: https://mlsanity.com/paper/2609.30950/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 34 references that are in the map.
- First posted on arXiv 25 Sep 2026.

## Builds on (most cited first, 28 of 28)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024)
- [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://mlsanity.com/paper/2404.00456/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024)
- [Kimi Linear: An Expressive, Efficient Attention Architecture](https://mlsanity.com/paper/2510.26692/) (2025)
- [Microscaling Data Formats for Deep Learning](https://mlsanity.com/paper/2310.10537/) (2023)
- [TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate](https://mlsanity.com/paper/2504.19874/) (2025)
- [REAP the Experts: Why Pruning Prevails for One-Shot MoE compression](https://mlsanity.com/paper/2510.13999/) (2025)
- [Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](https://mlsanity.com/paper/2606.15007/) (2026)
- [Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](https://mlsanity.com/paper/2604.12374/) (2026)
- [Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling](https://mlsanity.com/paper/2512.02010/) (2025)
- [Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention](https://mlsanity.com/paper/2605.22791/) (2026)
- [Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Models](https://mlsanity.com/paper/2503.22879/) (2025)
- [KV Cache Transform Coding for Compact Storage in LLM Inference](https://mlsanity.com/paper/2511.01815/) (2025)
- [Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs](https://mlsanity.com/paper/2511.16664/) (2025)
- [KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks](https://mlsanity.com/paper/2606.03458/) (2026)
- [DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization](https://mlsanity.com/paper/2608.27513/) (2026)
- [OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond](https://mlsanity.com/paper/2605.19660/) (2026)
- [KVBuffer: IO-aware Serving for Linear Attention](https://mlsanity.com/paper/2605.19049/) (2026)
- [SSDi8: Accurate and Efficient 8-bit Quantization for State Space Duality](https://mlsanity.com/paper/2608.21952/) (2026)
- [DASC: Decay-Aware State Compression for Hybrid Linear-Attention Serving](https://mlsanity.com/paper/2608.30386/) (2026)
- [Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM](https://mlsanity.com/paper/2609.04098/) (2026)
- [KV Cache Compression Through the Lens of Transform Coding](https://mlsanity.com/paper/2608.14191/) (2026)
- [DeltaLog: Deferred Materialization of Recurrent States for Linear Attention Decoding](https://mlsanity.com/paper/2608.15533/) (2026)

## Abstract (arXiv)

Hybrid language models maintain fixed-size recurrent states, but existing quantizers typically use eight bits or more. Quantization errors persist according to channel decay rates. We derive distortion weights from the observability Gramian and combine them with normalized state ranges for mixed-precision bit allocation, without calibration data, rotation, or training. We also quantize decay rates logarithmically. With per-token state quantization, a four-bit mean payload reduces excess negative log-likelihood by factors of 3.3--27.9 relative to the best of seven baselines across three hybrid models; metadata costs vary. At six bits, negative log-likelihood differs from the FP32-state baseline by less than 0.005 nats. Ablations separate gains from variable bit widths, decay weighting, and range normalization. With less frequent write-backs, gains diminish and depend on the model and budget.