mlsanityData to 7 Oct 2026

Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs

arXiv 2310.01801 ↗First posted 3 Oct 2023cs.CLcited by 466

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

In this study, we introduce adaptive KV cache compression, a plug-and-play method that reduces the memory footprint of generative inference for Large Language Models (LLMs). Different from the conventional KV cache that retains key and value vectors for all context tokens, we conduct targeted profiling to discern the intrinsic structure of attention modules. Based on the recognized structure, we then construct the KV cache in an adaptive manner: evicting long-range contexts on attention heads emphasizing local contexts, discarding non-special tokens on attention heads centered on special tokens, and only employing the standard KV cache for attention heads that broadly attend to all tokens. Moreover, with the lightweight attention profiling used to guide the construction of the adaptive KV cache, FastGen can be deployed without resource-intensive fine-tuning or re-training. In our experiments across various asks, FastGen demonstrates substantial reduction on GPU memory consumption with negligible generation quality loss. We will release our code and the compatible CUDA kernel for reproducibility.

Key facts. Cited by 466 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 39 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

29 of 39 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  2. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  9. 2023LIMA: Less Is More for Alignmentcited by 1,080
  10. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  11. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  12. 2019Are Sixteen Heads Really Better than One?cited by 803
15 more
  1. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  2. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  3. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  4. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  5. 2019Revealing the Dark Secrets of BERTcited by 300
  6. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  7. 2019Structured Pruning of Large Language Modelscited by 201
  8. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  9. 2021Learned Token Pruning for Transformerscited by 138
  10. 2017Skip RNN: Learning to Skip State Updates in Recurrent Neural Networkscited by 49
  11. 2022A Simple Hash-Based Early Exiting Approach For Language Understanding and Generationcited by 32
  12. 2022Transkimmer: Transformer Learns to Layer-wise Skimcited by 21
  13. 2019Neural Speed Reading with Structural-Jump-LSTMcited by 17
  14. 2017Neural Speed Reading via Skim-RNNcited by 14
  15. 2022Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selectioncited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 139; 2025: 173; 2026: 150.

466 papers.

Sort:

By area:

  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  2. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  3. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  4. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  5. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  6. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  7. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  8. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  9. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  10. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  11. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  12. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
38 more
  1. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  2. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  3. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  4. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  5. 2024Delta-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformerscited by 169
  6. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  7. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  8. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  9. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 148
  10. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 140
  11. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  12. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  13. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  14. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  15. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  16. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  17. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  18. 2025Why do LLMs attend to the first token?cited by 122
  19. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  20. 2024Transformers are Multi-State RNNscited by 120
  21. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  22. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  23. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  24. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  25. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  26. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
  27. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  28. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  29. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  30. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  31. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 96
  32. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 94
  33. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 93
  34. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  35. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  36. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 87
  37. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 86
  38. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 84

50 most cited of 466 All 466 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  2. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
  3. 2025ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMs
  4. 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
  5. 2026CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration
  6. 2026SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
  7. 2026ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs
  8. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression