mlsanityData to 14 Oct 2026

Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time

arXiv 2305.17118 ↗First posted 26 May 2023cs.LG cs.CLcited by 439

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language models(LLMs) have sparked a new wave of exciting AI applications. Hosting these models at scale requires significant memory resources. One crucial memory bottleneck for the deployment stems from the context window. It is commonly recognized that model weights are memory hungry; however, the size of key-value embedding stored during the generation process (KV cache) can easily surpass the model size. The enormous size of the KV cache puts constraints on the inference batch size, which is crucial for high throughput inference workload. Inspired by an interesting observation of the attention scores, we hypothesize the persistence of importance: only pivotal tokens, which had a substantial influence at one step, will significantly influence future generations. Based on our empirical verification and theoretical analysis around this hypothesis, we propose Scissorhands, a system that maintains the memory usage of the KV cache at a fixed budget without finetuning the model. In essence, Scissorhands manages the KV cache by storing the pivotal tokens with a higher probability. We validate that Scissorhands reduces the inference memory usage of the KV cache by up to 5X without compromising model quality. We further demonstrate that Scissorhands can be combined with 4-bit quantization, traditionally used to compress model weights, to achieve up to 20X compression.

Key facts. Cited by 439 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 30 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

24 of 30 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  11. 2020Reformer: The Efficient Transformercited by 1,668
  12. 2020Rethinking Attention with Performerscited by 1,601
12 more
  1. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  5. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  6. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  8. 2022Efficiently Scaling Transformer Inferencecited by 539
  9. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  10. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  11. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 130
  12. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 127; 2025: 118; 2026: 186.

439 papers.

Sort:

By area:

  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  2. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  4. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  5. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  6. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  7. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 348
  8. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 294
  9. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  10. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  11. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
  12. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
38 more
  1. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 217
  2. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  3. 2023Efficient Large Language Models: A Surveycited by 186
  4. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 169
  5. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  6. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  7. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 147
  8. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 143
  9. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  10. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 134
  11. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 133
  12. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 131
  13. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  14. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  15. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  16. 2024Transformers are Multi-State RNNscited by 120
  17. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  18. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
  19. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  20. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  21. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 102
  22. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 101
  23. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 100
  24. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 98
  25. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 97
  26. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 96
  27. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  28. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 95
  29. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
  30. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 89
  31. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  32. 2023Punica: Multi-Tenant LoRA Servingcited by 83
  33. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 76
  34. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  35. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matterscited by 73
  36. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compressioncited by 72
  37. 2024PruneVid: Visual Token Pruning for Efficient Video Large Language Modelscited by 67
  38. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66

50 most cited of 439 All 439 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Draft-based Approximate Inference for LLMs
  2. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs
  3. 2026Joint Encoding of KV-Cache Blocks for Scalable LLM Serving
  4. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  5. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
  6. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
  7. 2025ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMs
  8. 2026Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation