mlsanityData to 14 Oct 2026

Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving

arXiv 2607.05399 ↗First posted 3 May 2026cs.CL cs.AI cs.DCcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language model serving is increasingly limited by KV-cache growth under long-context workloads, yet existing KV-cache compression techniques are difficult to compare because they were evaluated on different models, tasks, budgets, and serving stacks. This paper presents a workload-aware benchmark of representative KV-cache optimization mechanisms spanning quantization, pruning, and merging, including KIVI, TurboQuant, SnapKV, and CaM, evaluated on LongBench-style multi-document QA, single-document QA, few-shot learning, and summarization workloads using Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3. The benchmark measures task quality, mean output throughput, mean time-to-first-token, and realized compression ratio across context-length buckets. The results show that the compression ratio alone is a poor predictor of end-to-end performance. KIVI4 provides the most stable quality across models, SnapKV delivers the strongest long-context throughput, and CaM yields large gains on selected QA workloads but exhibits substantial workload sensitivity in both quality and realized compression ratio. These findings motivate workload-aware selection of KV-cache mechanisms rather than one-size-fits-all compression and provide deployment guidance for long-context serving systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 30 of its 48 references that are in the map. First posted on arXiv 3 May 2026.

Builds on

30 of 48 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution1 Oct
  2. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review8 Aug
  3. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider3 Jun
  4. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference11 Mar
  5. 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
  6. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing24 Oct
  7. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy4 Oct
  8. 2024Palu: Compressing KV-Cache with Low-Rank Projection30 Jul
  9. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches1 Jul
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  11. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  12. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct

Other references · 18

  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  5. 2022Token Merging: Your ViT But Fastercited by 1,021
  6. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  9. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
  10. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  11. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  12. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
6 more
  1. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  2. 2023Compressed Context Memory For Online Language Model Interactioncited by 43
  3. 2024Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMscited by 30
  4. 2024MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decodingcited by 21
  5. 2025LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillationcited by 14
  6. 2024Are Large Language Models the New Interface for Data Pipelines?cited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models
  2. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
  3. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques
  4. 2026KV Cache Offloading for Context-Intensive Tasks
  5. 2025Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?
  6. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scores
  7. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving
  8. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs