mlsanityData to 14 Oct 2026

RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache

arXiv 2605.08317 ↗First posted 8 May 2026cs.LG cs.AIcited by 4

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Large language models (LLMs) have shown strong performance across diverse tasks, but their inference with long input contexts is bottlenecked by memory size and bandwidth. The Key-Value (KV) cache size grows linearly with sequence length and needs to be re-read from off-chip high-bandwidth memory (HBM) to on-chip memory at every decoding step, resulting in memory-bound inference. Existing methods reduce the cache by either eviction or quantization, but typically treat the two in isolation. In this paper, we cast KV cache compression as a rate-distortion problem, under which eviction and quantization are two end-points of the same bit allocation scheme. This exposes the need to optimize them jointly, motivating our method, RDKV (Rate-Distortion KV cache compression). RDKV derives the weight of each token or channel from the distortion that compression induces on the attention computation. Based on these weights, it assigns each token or channel a bit-width ranging from full precision down to zero bits guided by reverse water-filling, applied once after the prefilling stage. Experiments on LongBench, RULER, and InfiniteBench show that RDKV outperforms the best evaluated baseline by 9.1% on average. On LongBench it recovers 97.81% of full-cache accuracy with only 2.48% cache retention. Compared with full-cache FlashAttention-2 decoding, it achieves 4.5x decode speedup and 1.9x peak memory reduction with 128K context length, while maintaining comparable performance.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 40 of its 49 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

40 of 49 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference6 Feb
  2. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification23 May
  3. 2024QAQ: Quality Adaptive Quantization for LLM KV Cache7 Mar
  4. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  5. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  7. 2025Qwen3 Technical Reportcited by 8,820
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
23 more
  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  2. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  5. 2022Efficiently Scaling Transformer Inferencecited by 539
  6. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  7. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  8. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  9. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
  10. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  11. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 101
  12. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 74
  13. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 68
  14. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferencescited by 58
  15. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Timecited by 43
  16. 2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspectivecited by 26
  17. 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Modelscited by 24
  18. 2025CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Evictioncited by 13
  19. 2025KV Cache Compression for Inference Efficiency in LLMs: A Reviewcited by 12
  20. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compressioncited by 10
  21. 2025OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inferencecited by 10
  22. 2025Optimal Brain Restoration for Joint Quantization and Sparsification of LLMscited by 6
  23. 2026ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attentioncited by 0
  2. 2026Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoningcited by 0
  3. 2026Mask-Guided KV Cache Eviction in Block Diffusion Language Modelscited by 0
  4. 2026Read What Matters: Query-Adaptive Quantization for KV Cachescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
  2. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  3. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization
  4. 2025H1B-KV: Hybrid One-Bit Caches for Memory-Efficient Large Language Model Inference
  5. 2026RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory
  6. 2026DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression
  7. 2026A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
  8. 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics