mlsanityData to 7 Oct 2026

KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches

arXiv 2407.01527 ↗First posted 1 Jul 2024cs.CLcited by 43

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long context capability is a crucial competency for large language models (LLMs) as it mitigates the human struggle to digest long-form texts. This capability enables complex task-solving scenarios such as book summarization, code assistance, and many more tasks that are traditionally manpower-intensive. However, transformer-based LLMs face significant challenges with long context input due to the growing size of the KV cache and the intrinsic complexity of attending to extended inputs; where multiple schools of efficiency-driven approaches - such as KV cache quantization, token dropping, prompt compression, linear-time sequence models, and hybrid architectures - have been proposed to produce efficient yet long context-capable models. Despite these advancements, no existing work has comprehensively benchmarked these methods in a reasonably aligned environment. In this work, we fill this gap by providing a taxonomy of current methods and evaluating 10+ state-of-the-art approaches across seven categories of long context tasks. Our work reveals numerous previously unknown phenomena and offers insights - as well as a friendly workbench - for the future development of long context-capable LLMs. The source code is available at https://github.com/henryzhongsc/longctx_bench.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 58 of its 71 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

58 of 71 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention21 May
  2. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  3. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  4. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
  5. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  6. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 52

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
38 more
  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  2. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  3. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  4. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  8. 2022Galactica: A Large Language Model for Sciencecited by 736
  9. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  10. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  11. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  12. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  13. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  14. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  15. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  16. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  17. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  18. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  19. 2023Adapting Language Models to Compress Contextscited by 350
  20. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  21. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  22. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  23. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  24. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  25. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  26. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  27. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  28. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  29. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  30. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  31. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 100
  32. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 96
  33. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 74
  34. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  35. 2024RecurrentGemma: Moving Past Transformers for Efficient Open Language Modelscited by 61
  36. 2024Learning to Compress Prompt in Natural Language Formatscited by 54
  37. 2024Contextual Position Encoding: Learning to Count What's Importantcited by 53
  38. 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overheadcited by 47

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 21; 2026: 13.

43 papers.

Sort:

By area:

  1. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  2. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  3. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  4. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compressioncited by 41
  5. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  6. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  7. 2025CommVQ: Commutative Vector Quantization for KV Cache Compressioncited by 25
  8. 2024SCOPE: Optimizing Key-Value Cache Compression in Long-context Generationcited by 20
  9. 2025Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMscited by 20
  10. 2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compressioncited by 16
  11. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devicescited by 15
  12. 2025Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Querycited by 15
31 more
  1. 2025The Pitfalls of KV Cache Compressioncited by 15
  2. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inferencecited by 13
  3. 2025Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatchcited by 13
  4. 2024LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptationcited by 11
  5. 2025KV Cache Transform Coding for Compact Storage in LLM Inferencecited by 11
  6. 2025ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMscited by 10
  7. 2025SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMscited by 9
  8. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Formatcited by 8
  9. 2025RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decodingcited by 8
  10. 2025LagKV: Lag-Relative Information of the KV Cache Tells Which Tokens Are Importantcited by 7
  11. 2024ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiencycited by 6
  12. 2024A Survey of RWKVcited by 5
  13. 2026MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inferencecited by 5
  14. 2025Cost-Optimal Grouped-Query Attention for Long-Context Modelingcited by 4
  15. 2026Scout Before You Attend: Sketch-and-Walk Sparse Attention for Efficient LLM Inferencecited by 4
  16. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefillingcited by 3
  17. 2025ILRe: Intermediate Layer Retrieval for Context Compression in Causal Language Modelscited by 3
  18. 2026KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compressioncited by 2
  19. 2025Self-ensemble: Mitigating Confidence Mis-calibration for Large Language Modelscited by 1
  20. 2025Systematic Evaluation of Optimization Techniques for Long-Context Language Modelscited by 1
  21. 2026Screening Is Enoughcited by 1
  22. 2026Forget Without Compromise: Nexus Sampling for Streaming KV-Cache Eviction Under Fixed Budgetscited by 1
  23. 2026KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Modelscited by 1
  24. 2025Task-Oriented Communications for 3D Scene Representation: Balancing Timeliness and Fidelitycited by 0
  25. 2026Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Studycited by 0
  26. 2026Latent Phase-Shift Rollback: Inference-Time Error Correction via Residual Stream Monitoring and KV-Cache Steeringcited by 0
  27. 2026Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Servingcited by 0
  28. 2026VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMscited by 0
  29. 2026HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decodingcited by 0
  30. 2026Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoningcited by 0
  31. 2026DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
  2. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques
  3. 2025KVComp: A High-Performance, LLM-Aware, Lossy Compression Framework for KV Cache
  4. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  5. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
  6. 2025Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?
  7. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression
  8. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head