# ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals

arXiv: https://arxiv.org/abs/2610.10381 · HTML: https://mlsanity.com/paper/2610.10381/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 17 of its 37 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Placed in

- Efficient LLM inference and serving / [KV cache quantization](https://mlsanity.com/inference/kv-cache-quantization/)

## Builds on (most cited first, 17 of 17)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [Hierarchical Reasoning Model](https://mlsanity.com/paper/2506.21734/) (2025)
- [Scaling Latent Reasoning via Looped Language Models](https://mlsanity.com/paper/2510.25741/) (2025)
- [Less is More: Recursive Reasoning with Tiny Networks](https://mlsanity.com/paper/2510.04871/) (2025)
- [Parallel Loop Transformer for Efficient Test-Time Computation Scaling](https://mlsanity.com/paper/2510.24824/) (2025)
- [SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers](https://mlsanity.com/paper/2609.01343/) (2026)
- [Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models](https://mlsanity.com/paper/2605.07721/) (2026)
- [OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization](https://mlsanity.com/paper/2605.17757/) (2026)
- [Thinking with Looped Flows](https://mlsanity.com/paper/2609.11801/) (2026)
- [LoopQ: Quantization for Recursive Transformers](https://mlsanity.com/paper/2605.16343/) (2026)
- [RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory](https://mlsanity.com/paper/2609.03379/) (2026)
- [Output-Aware Rotation for INT2 KV-Cache Quantization](https://mlsanity.com/paper/2608.02691/) (2026)
- [LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers](https://mlsanity.com/paper/2609.17184/) (2026)
- [FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates](https://mlsanity.com/paper/2609.29812/) (2026)

## Abstract (arXiv)

Looped Transformers improve parameter efficiency by repeatedly applying shared Transformer blocks over multiple recurrent loops, increasing computational depth without increasing the parameter count. However, KV cache memory still scales with the number of loops, becoming a key memory bottleneck that limits batch size and inference throughput. KV cache quantization can alleviate this bottleneck, but existing methods often suffer substantial accuracy degradation at aggressive low-precision regimes. We observe that looped Transformers offer a unique opportunity: KV states across loops are highly similar. Based on this observation, we propose ResidualQuant, which uses the final-loop KV states as a reference and represents the remaining loops with low-precision residuals. Our method further combines least-square scaling and rotations applied to the residuals, as well as loop-wise mixed precision, to enable accurate quantization down to INT2 while retaining efficient reconstruction. Across multiple looped Transformer models and mathematical reasoning and code generation benchmarks, ResidualQuant consistently improves the accuracy-memory tradeoff over state-of-the-art rotation-based KV quantization. In particular, our method retains accuracy close to BF16 under mixed-precision settings while reducing theoretical KV storage by 80.7%, achieving up to 13.0% higher accuracy than the rotation-based baseline at the same memory budget. On an RTX 5090, the reduced KV memory traffic improves fixed-batch decode throughput by up to 2.73x, while the smaller memory footprint enables up to 2x larger batches, improving peak throughput by up to 4.15x.