RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference
On the map: Efficient LLM inference and serving → KV cache quantization
Abstract (arXiv)
Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that are computationally expensive and biased. To address these limitations, we propose RaBitQCache, a novel sparse attention framework that utilizes randomized rotated binary quantization and high-throughput binary-INT4 arithmetic to efficiently estimate attention weights. Our proxy score serves as an unbiased estimator with a proven error bound, enabling adaptive Top-p retrieval that dynamically adjusts the token budget based on actual attention sparsity. We further implement a hardware-aware system with asynchronous pipelining and lazy updates to mask overhead. Evaluations demonstrate that RaBitQCache significantly accelerates inference and reduces memory I/O while preserving generation quality compared to state-of-the-art baselines. Code is available at https://github.com/Sakuraaa0/RaBitQCache.git.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 40 of its 47 references that are in the map. First posted on arXiv 30 Jun 2026.
Builds on
40 of 47 references.
Closest ancestors · 3
Same direction, newest first
- 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inference1 Jul
- 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
- 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Serving29 Oct
Other references · 37
- 2017Attention Is All You Needcited by 60,361
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2023GPT-4 Technical Reportcited by 18,396
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
25 more
- 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
- 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
- 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
- 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
- 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
- 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
- 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
- 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
- 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
- 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
- 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
- 2025Qwen2.5-1M Technical Reportcited by 164
- 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
- 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
- 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
- 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
- 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 101
- 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
- 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 49
- 2024RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Searchcited by 48
- 2024Post-Training Sparse Attention with Double Sparsitycited by 45
- 2026Beyond Speedup -- Utilizing KV Cache for Sampling and Reasoningcited by 3
- 2017Generic LSH Families for the Angular Distance Based on Johnson-Lindenstrauss Projections and Feature Hashing LSHcited by 1
- 2025VanGogh: A Unified Multimodal Diffusion-based Framework for Video Colorizationcited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Output-Aware Rotation for INT2 KV-Cache Quantization
- 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
- 2026Runtime-Certified Bounded-Error Quantized Attention
- 2025Rectified Sparse Attention
- 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction
- 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling
- 2025Efficient Low Rank Attention for Long-Context Inference in Large Language Models
- 2025A2ATS: Retrieval-Based KV Cache Reduction via Windowed Rotary Position Embedding and Query-Aware Vector Quantization