UltraQuant: 4-bit KV Caching for Context-Heavy Agents
Not on the map. Why?
Abstract (arXiv)
Context-heavy agents place substantial pressure on the key-value (KV) cache: long prefixes are reused across many short turns, while concurrency determines whether the serving system can keep GPUs utilized. We study 4-bit KV-cache compression for this setting, using TurboQuant-style rotation and codebook quantization as a quality anchor and vLLM FP8 KV caching as the deployment anchor. We report three contributions. First, we frame 4-bit KV caching around multi-round agent workloads where task quality, cache residency, and serving throughput must be measured jointly. Second, we describe the practical design choices needed to make the 4-bit path robust, including asymmetric K/V treatment, Walsh-Hadamard rotation, QJL removal, and block-scale variants. Third, we present serving optimizations on AMD GPUs, including optimized decode-attention kernels and UltraQuant, an FP4 approximation path that uses FP8 queries, FP4 KV tensors, UE8M0 group scales, and native scaled-MFMA support on CDNA4. On an adaptive-SLO replay of production Claude Code traces, UltraQuant sustains 2.71x (MiniMax-M2.5) and 4.38x (Qwen3-235B) the qualified-request throughput of the BF16 baseline, matching or exceeding hardware FP8 KV while using half the KV bytes. UltraQuant delivers its largest gains in long-context, high-concurrency, memory-constrained serving regimes.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 40 references that are in the map. First posted on arXiv 18 Jun 2026.
Builds on
18 of 40 references.
References · 18
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
- 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
- 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
- 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
- 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
- 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
6 more
- 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
- 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
- 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
- 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overheadcited by 47
- 2024Fast Matrix Multiplications for Lookup Table-Quantized LLMscited by 27
- 2026Statistical Inference and Quality Measures of KV Cache Quantisations Inspired by TurboQuantcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026The KV Cache Is the New Memory Wall
- 2026SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving
- 2026KV-COBRA: KV Cache Compression via Co-Optimized Bit-Rank Allocation
- 2026TurboAngle: Near-Lossless KV Cache Compression via Uniform Angle Quantization
- 2026Every Cache Entry Earns Its Place: Global Allocation of Resolution and Coverage for KV Cache Compression
- 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
- 2026Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference
- 2026More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving