mlsanityData to 14 Oct 2026

KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments

arXiv 2504.15364 ↗First posted 21 Apr 2025cs.AIcited by 51

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

We demonstrate that geometrically distinctive keys during LLM inference tend to have high attention scores. Based on the phenomenon we propose KeyDiff, a training-free KV cache eviction method based solely on key similarity. Unlike other KV cache eviction methods, KeyDiff can process arbitrarily long prompts within strict resource constraints and efficiently generate responses. We provide a theoretical basis for KeyDiff by relating key diversity with attention scores. These results imply KeyDiff can efficiently identify the most important tokens to retain. Notably KeyDiff does not rely on attention scores, allowing the use of optimized attention mechanisms like FlashAttention. Under a strict memory allowance, we demonstrate the effectiveness of KeyDiff for the Llama and Qwen model families by observing a performance gap of less than 0.04% with 8K cache budget (~23% KV cache reduction) from the non-evicting baseline on LongBench for Llama 3.1-8B and Llama 3.2-3B. We also observe near baseline performance for Deepseek-R1-Distill-Llama-8B on the Math500 reasoning benchmark and decrease end-to-end inference latency by up to 30% compared to the other token-eviction methods.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 41 of its 46 references that are in the map. First posted on arXiv 21 Apr 2025.

Builds on

41 of 46 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head30 Sep
  2. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul
  3. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression17 Jun
  4. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  5. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  7. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
  8. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  12. 2020Longformer: The Long-Document Transformercited by 3,413
19 more
  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2020GLU Variants Improve Transformercited by 1,804
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  7. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  8. 2024Massive Activations in Large Language Modelscited by 294
  9. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  10. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  11. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  12. 2025Why do LLMs attend to the first token?cited by 127
  13. 2024Transformers are Multi-State RNNscited by 120
  14. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  15. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 82
  16. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 76
  17. 2024Anisotropy Is Inherent to Self-Attention in Transformerscited by 63
  18. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  19. 2024Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMscited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 39.

51 papers.

Sort:

By area:

  1. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 74
  2. 2025InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understandingcited by 58
  3. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 58
  4. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 19
  5. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scorescited by 17
  6. 2025The Pitfalls of KV Cache Compressioncited by 15
  7. 2026Learning to Evict from Key-Value Cachecited by 14
  8. 2026Self-Compacting Language Model Agentscited by 11
  9. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Servingcited by 10
  10. 2026LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generationcited by 10
  11. 2025KaVa: Latent Reasoning via Compressed KV-Cache Distillationcited by 8
  12. 2025PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inferencecited by 7
38 more
  1. 2025EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environmentscited by 7
  2. 2026IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inferencecited by 7
  3. 2026Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Evictioncited by 6
  4. 2026MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inferencecited by 6
  5. 2026Rethinking KV Cache Eviction via a Unified Information-Theoretic Objectivecited by 5
  6. 2026A Simple Plug-in for Improving Eviction-Based KV Cache Compressioncited by 4
  7. 2026AVIS: Adaptive Test-Time Scaling for Vision-Language Modelscited by 4
  8. 2026QUOKA: Query-Oriented KV Selection For Efficient LLM Prefillcited by 3
  9. 2026ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMscited by 2
  10. 2026ManifoldKV: Training-Free KV Cache Compression via Euclidean Outlier Detectioncited by 2
  11. 2026Neural Garbage Collection: Learning to Forget while Learning to Reasoncited by 2
  12. 2026HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compressioncited by 2
  13. 2026KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compressioncited by 2
  14. 2025Efficient On-Device Agents via Adaptive Context Managementcited by 1
  15. 2026Fast Forward: Accelerating LLM Prefill with Predictive FFN Sparsitycited by 1
  16. 2026CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inferencecited by 1
  17. 2026KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Modelscited by 1
  18. 2026Random Attention: Rethinking KV Cache Eviction for Efficient Reasoningcited by 1
  19. 2026EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstructioncited by 1
  20. 2025Leveraging KV Similarity for Online Structured Pruning in LLMscited by 0
  21. 2026ConFu: Contemplate the Future for Better Speculative Samplingcited by 0
  22. 2026Reducing Peak Memory Usage for Modern Multimodal Large Language Model Pipelinescited by 0
  23. 2026Head-Aware Key-Value Compression for Efficient Autoregressive Image Generationcited by 0
  24. 2026Express Language Modelingcited by 0
  25. 2026How Query Visibility Changes KV-Cache Compression Rankings: A Matched-Budget Auditcited by 0
  26. 2026CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agentscited by 0
  27. 2026DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inferencecited by 0
  28. 2026Sigmoid Attention as a Better Substrate for Learned KV Cache Evictioncited by 0
  29. 2026BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inferencecited by 0
  30. 2026ECOKV: Geometry-Aware KV Cache Eviction via Complementary Diversity Metricscited by 0
  31. 2026Jacap: Robust KV Cache Eviction via Jacobian-Based Nonlinear Information Capacity Preservationcited by 0
  32. 2026ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMscited by 0
  33. 2026DeltaS: Reading the Gated Linear Attention State for KV Cache Eviction in Streaming Videocited by 0
  34. 2026CORE: COverage CAlibration and Evicted-Mass REdistribution for KV Cachecited by 0
  35. 2026TPBench: A Turning-Point Benchmark for Dialogue Compressioncited by 0
  36. 2026AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVDcited by 0
  37. 2026A Self-Pruning Transformer: Extreme KV-Cache Compression with Universal Attentioncited by 0
  38. 2026Rehearse Everything, Remember Nothing: Attic-KV Rehearses What Will Be Readcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  2. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
  3. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
  4. 2026Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction
  5. 2026CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
  6. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
  7. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
  8. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time