mlsanityData to 7 Oct 2026

RazorAttention: Efficient KV Cache Compression Through Retrieval Heads

arXiv 2407.15891 ↗First posted 22 Jul 2024cs.LG cs.CLcited by 99

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The memory and computational demands of Key-Value (KV) cache present significant challenges for deploying long-context language models. Previous approaches attempt to mitigate this issue by selectively dropping tokens, which irreversibly erases critical information that might be needed for future queries. In this paper, we propose a novel compression technique for KV cache that preserves all token information. Our investigation reveals that: i) Most attention heads primarily focus on the local context; ii) Only a few heads, denoted as retrieval heads, can essentially pay attention to all input tokens. These key observations motivate us to use separate caching strategy for attention heads. Therefore, we propose RazorAttention, a training-free KV cache compression algorithm, which maintains a full cache for these crucial retrieval heads and discards the remote tokens in non-retrieval heads. Furthermore, we introduce a novel mechanism involving a "compensation token" to further recover the information in the dropped tokens. Extensive evaluations across a diverse set of large language models (LLMs) demonstrate that RazorAttention achieves a reduction in KV cache size by over 70% without noticeable impacts on performance. Additionally, RazorAttention is compatible with FlashAttention, rendering it an efficient and plug-and-play solution that enhances LLM inference efficiency without overhead or retraining of the original model.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 19 of its 28 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

19 of 28 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  3. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 16

  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  2. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  3. 2019Root Mean Square Layer Normalizationcited by 1,425
  4. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  6. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  7. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  9. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  12. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
4 more
  1. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  2. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  3. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  4. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 38; 2026: 48.

99 papers.

Sort:

By area:

  1. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  2. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  4. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  5. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 96
  6. 2024Attention Heads of Large Language Models: A Surveycited by 65
  7. 2025InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understandingcited by 54
  8. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  9. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compressioncited by 41
  10. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 38
  11. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  12. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 35
38 more
  1. 2025Cognitive Memory in Large Language Modelscited by 28
  2. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  3. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inferencecited by 19
  4. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scorescited by 16
  5. 2025Intrinsic Entropy of Context Length Scaling in LLMscited by 14
  6. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inferencecited by 14
  7. 2025Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulationcited by 13
  8. 2026Past- and Future-Informed KV Cache Policy with Salience Estimation in Autoregressive Video Diffusioncited by 13
  9. 2025KV Cache Compression for Inference Efficiency in LLMs: A Reviewcited by 12
  10. 2024LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptationcited by 11
  11. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compressioncited by 10
  12. 2025Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inferencecited by 10
  13. 2025Which Heads Matter for Reasoning? RL-Guided KV Cache Compressioncited by 10
  14. 2025MEPIC: Memory Efficient Position Independent Caching for LLM Servingcited by 10
  15. 2024A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contextscited by 9
  16. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloadingcited by 9
  17. 2026ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothingcited by 9
  18. 2025CentroidKV: Efficient Long-Context LLM Inference via KV Cache Clusteringcited by 8
  19. 2025OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inferencecited by 8
  20. 2025Memory in Large Language Models: Mechanisms, Evaluation and Evolutioncited by 8
  21. 2025MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoningcited by 8
  22. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 7
  23. 2025LagKV: Lag-Relative Information of the KV Cache Tells Which Tokens Are Importantcited by 7
  24. 2025RainFusion: Adaptive Video Generation Acceleration via Multi-Dimensional Visual Redundancycited by 7
  25. 2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inferencecited by 7
  26. 2024Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQAcited by 6
  27. 2025SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruningcited by 6
  28. 2026MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answeringcited by 6
  29. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttentioncited by 6
  30. 2025Unveiling Simplicities of Attention: Adaptive Long-Context Head Identificationcited by 5
  31. 2025Compression Barriers for Autoregressive Transformerscited by 5
  32. 2025EdgeInfinite: A Memory-Efficient Infinite-Context Transformer for Edge Devicescited by 5
  33. 2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generationcited by 5
  34. 2026LycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decodingcited by 5
  35. 2026Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMscited by 5
  36. 2026When Less Latent Leads to Better Relay: Information-Preserving Compression for Latent Multi-Agent LLM Collaborationcited by 5
  37. 2025Task-KV: Task-aware KV Cache Optimization via Semantic Differentiation of Attention Headscited by 4
  38. 2025CAFE: Retrieval Head-based Coarse-to-Fine Information Seeking to Enhance Multi-Document QA Capabilitycited by 4

50 most cited of 99 All 99 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation
  2. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques
  3. 2026KV2: A Self-Refining KV Cache
  4. 2026More Than a Quick Glance: Overcoming the Greedy Bias in KV-Cache Compression
  5. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head
  6. 2024EMS: Adaptive Evict-then-Merge Strategy for Head-wise KV Cache Compression Based on Global-Local Importance
  7. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
  8. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression