mlsanityData to 14 Oct 2026

SCOPE: Optimizing Key-Value Cache Compression in Long-context Generation

arXiv 2412.13649 ↗First posted 18 Dec 2024cs.CLcited by 20

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Key-Value (KV) cache has become a bottleneck of LLMs for long-context generation. Despite the numerous efforts in this area, the optimization for the decoding phase is generally ignored. However, we believe such optimization is crucial, especially for long-output generation tasks based on the following two observations: (i) Excessive compression during the prefill phase, which requires specific full context impairs the comprehension of the reasoning task; (ii) Deviation of heavy hitters occurs in the reasoning tasks with long outputs. Therefore, SCOPE, a simple yet efficient framework that separately performs KV cache optimization during the prefill and decoding phases, is introduced. Specifically, the KV cache during the prefill phase is preserved to maintain the essential information, while a novel strategy based on sliding is proposed to select essential heavy hitters for the decoding phase. Memory usage and memory transfer are further optimized using adaptive and discontinuous strategies. Extensive experiments on LongGenBench show the effectiveness and generalization of SCOPE and its compatibility as a plug-in to other prefill-only KV compression methods.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 40 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

29 of 40 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  2. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul
  3. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption25 Jul
  4. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  5. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches1 Jul
  6. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  8. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 19

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  11. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  12. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
7 more
  1. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  2. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  3. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  4. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  5. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 89
  6. 2024Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMscited by 30
  7. 2024SEED: Accelerating Reasoning Tree Construction via Scheduled Speculative Decodingcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 12.

20 papers.

Sort:

By area:

  1. 2025LightThinker: Thinking Step-by-Step Compressioncited by 104
  2. 2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compressioncited by 16
  3. 2025ParallelMuse: Agentic Parallel Thinking for Deep Information Seekingcited by 14
  4. 2025Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generationcited by 11
  5. 2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inferencecited by 7
  6. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMscited by 7
  7. 2025GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Evictioncited by 6
  8. 2025EdgeInfinite: A Memory-Efficient Infinite-Context Transformer for Edge Devicescited by 5
  9. 2026LightThinker++: From Reasoning Compression to Memory Managementcited by 4
  10. 2026HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compressioncited by 2
  11. 2025CCF: A Context Compression Framework for Efficient Long-Sequence Language Modelingcited by 1
  12. 2026S3-Attention:Attention-Aligned Endogenous Retrieval for Memory-Bounded Long-Context Inferencecited by 1
8 more
  1. 2026UT-ACA: Uncertainty-Triggered Adaptive Context Allocation for Long-Context Inferencecited by 1
  2. 2026Adaptive Mass-Segmented KV Compression for Long-Context Reasoningcited by 1
  3. 2026Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generationcited by 0
  4. 2026ETC: Extreme Token Compression via Task-aware Visual Information Distillation in VLMscited by 0
  5. 2026Parser Transitions Already Reveal Structural Responsibility: KV Persistence for Structured Generationcited by 0
  6. 2026StepKV: Step-Aware KV Cache Compression for LLM Agentscited by 0
  7. 2026CORE: COverage CAlibration and Evicted-Mass REdistribution for KV Cachecited by 0
  8. 2026WakeKV: Reactive, Reversible KV Residency for Heads That Change Their Mindscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs
  2. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques
  3. 2025Hold Onto That Thought: Assessing KV Cache Compression On Reasoning
  4. 2026KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
  5. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head
  6. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  7. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  8. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference