mlsanityData to 7 Oct 2026

LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference

arXiv 2503.08879 ↗First posted 11 Mar 2025cs.CL cs.AI cs.LGcited by 18

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Efficient long-context inference is critical as large language models (LLMs) adopt context windows of ranging from 128K to 1M tokens. However, the growing key-value (KV) cache and the high computational complexity of attention create significant bottlenecks in memory usage and latency. In this paper, we find that attention in diverse long-context tasks exhibits sparsity, and LLMs implicitly "know" which tokens can be dropped or evicted at the head level after the pre-filling stage. Based on this insight, we propose Self-Attention Guided Eviction~(SAGE-KV), a simple and effective KV eviction cache method for long-context inference. After prefilling, our method performs a one-time top-k selection at both the token and head levels to compress the KV cache, enabling efficient inference with the reduced cache. Evaluations on LongBench and three long-context LLMs (Llama3.1-8B-Instruct-128k, Llama3-8B-Prolong-512k-Instruct, and Qwen2.5-7B-Instruct-128k) show that SAGE-KV maintains accuracy comparable to full attention while significantly improving efficiency. Specifically, SAGE-KV achieves 4x higher memory efficiency with improved accuracy over the static KV cache selection method StreamLLM, and 2x higher memory efficiency with better accuracy than the dynamic KV cache selection method Quest.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 19 of its 20 references that are in the map. First posted on arXiv 11 Mar 2025.

Builds on

19 of 20 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
  2. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  3. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  5. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  6. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 13

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  6. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  7. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  8. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  9. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  10. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  11. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  12. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
1 more
  1. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 11.

18 papers.

Sort:

By area:

  1. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 18
  2. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruningcited by 13
  3. 2025Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?cited by 10
  4. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Evictioncited by 10
  5. 2026LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Evictioncited by 4
  6. 2025SnapStream: Efficient Long Sequence Decoding on Dataflow Acceleratorscited by 2
  7. 2026Cross-Family Speculative Prefill: Training-Free Long-Context Compression with Small Draft Modelscited by 2
  8. 2026Prefix Sliding for efficient test-time scalingcited by 2
  9. 2026More Than a Quick Glance: Overcoming the Greedy Bias in KV-Cache Compressioncited by 1
  10. 2026Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inferencecited by 1
  11. 2026Monte Carlo Estimation for KV Cache Evictioncited by 1
  12. 2025PatternKV: Flattening KV Representation Expands Quantization Headroomcited by 0
6 more
  1. 2025APCE: Adaptive Progressive Context Expansion for Long Context Processingcited by 0
  2. 2025KV Cache Recycling to Expand Usable Context Capacity in Low Parameter LLMscited by 0
  3. 2026S3-Attention:Attention-Aligned Endogenous Retrieval for Memory-Bounded Long-Context Inferencecited by 0
  4. 2026UT-ACA: Uncertainty-Triggered Adaptive Context Allocation for Long-Context Inferencecited by 0
  5. 2026Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Servingcited by 0
  6. 2026Learning how to Forget: Fine-tuning for Long-Context Sparse Attentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  2. 2024In-context KV-Cache Eviction for LLMs via Attention-Gate
  3. 2026IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
  4. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time
  5. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
  6. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
  7. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices
  8. 2026Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM