mlsanityData to 7 Oct 2026

SnapKV: LLM Knows What You are Looking for Before Generation

arXiv 2404.14469 ↗First posted 22 Apr 2024cs.CL cs.AIcited by 848

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large Language Models (LLMs) have made remarkable progress in processing extensive contexts, with the Key-Value (KV) cache playing a vital role in enhancing their performance. However, the growth of the KV cache in response to increasing input length poses challenges to memory and time efficiency. To address this problem, this paper introduces SnapKV, an innovative and fine-tuning-free approach that efficiently minimizes KV cache size while still delivering comparable performance in real-world applications. We discover that each attention head in the model consistently focuses on specific prompt attention features during generation. Meanwhile, this robust pattern can be obtained from an 'observation' window located at the end of the prompts. Drawing on this insight, SnapKV automatically compresses KV caches by selecting clustered important KV positions for each attention head. Our approach significantly reduces the growing computational overhead and memory footprint when processing long input sequences. Specifically, SnapKV achieves a consistent decoding speed with a 3.6x increase in generation speed and an 8.2x enhancement in memory efficiency compared to the baseline when processing inputs of 16K tokens. At the same time, it maintains comparable performance to the baseline models across 16 long sequence datasets. Moreover, SnapKV can process up to 380K context tokens on a single A100-80GB GPU using HuggingFace implementation with minor changes, exhibiting only a negligible accuracy drop in the Needle-in-a-Haystack test. Further comprehensive studies suggest SnapKV's potential for practical applications.

Key facts. Cited by 848 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 37 references that are in the map. First posted on arXiv 22 Apr 2024.

Builds on

25 of 37 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  3. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 22

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  11. 2022In-context Learning and Induction Headscited by 930
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
10 more
  1. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  2. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  3. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  4. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  5. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  6. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 139
  7. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  8. 2020Extractive Opinion Summarization in Quantized Transformer Spacescited by 76
  9. 2024Recurrent Drafter for Fast Speculative Decoding in Large Language Modelscited by 35
  10. 2023Overview of BioASQ 2023: The eleventh BioASQ challenge on Large-Scale Biomedical Semantic Indexing and Question Answeringcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 80; 2025: 257; 2026: 511.

848 papers.

Sort:

By area:

  1. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  2. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  3. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  4. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  5. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  6. 2025MIRIX: Multi-Agent Memory System for LLM-Based Agentscited by 160
  7. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  8. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  9. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  10. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  11. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
  12. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
38 more
  1. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  2. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 112
  3. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  4. 2025LightThinker: Thinking Step-by-Step Compressioncited by 104
  5. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 96
  6. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 94
  7. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 93
  8. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  9. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  10. 2025StreamingVLM: Real-Time Understanding for Infinite Video Streamscited by 89
  11. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 84
  12. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  13. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
  14. 2025Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compressioncited by 72
  15. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  16. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  17. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 66
  18. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compressioncited by 65
  19. 2025Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Evictioncited by 64
  20. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsificationcited by 59
  21. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  22. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferencescited by 59
  23. 2025PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoningcited by 58
  24. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 57
  25. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 56
  26. 2025InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understandingcited by 54
  27. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  28. 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Modelscited by 52
  29. 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Modelscited by 52
  30. 2025M+: Extending MemoryLLM with Scalable Long-Term Memorycited by 51
  31. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 50
  32. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 49
  33. 2025Mem-α: Learning Memory Construction via Reinforcement Learningcited by 49
  34. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  35. 2024Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysiscited by 44
  36. 2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44
  37. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  38. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMscited by 42

50 most cited of 848 All 848 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Effectively Compress KV Heads for LLM
  2. 2024Beyond KV Caching: Shared Attention for Efficient LLMs
  3. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  4. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
  5. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
  7. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention
  8. 2024In-context KV-Cache Eviction for LLMs via Attention-Gate