mlsanityData to 14 Oct 2026

A Survey on Large Language Model Acceleration based on KV Cache Management

arXiv 2412.19442 ↗First posted 27 Dec 2024cs.AI cs.DCcited by 131

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) have revolutionized a wide range of domains such as natural language processing, computer vision, and multi-modal tasks due to their ability to comprehend context and perform logical reasoning. However, the computational and memory demands of LLMs, particularly during inference, pose significant challenges when scaling them to real-world, long-context, and real-time applications. Key-Value (KV) cache management has emerged as a critical optimization technique for accelerating LLM inference by reducing redundant computations and improving memory utilization. This survey provides a comprehensive overview of KV cache management strategies for LLM acceleration, categorizing them into token-level, model-level, and system-level optimizations. Token-level strategies include KV cache selection, budget allocation, merging, quantization, and low-rank decomposition, while model-level optimizations focus on architectural innovations and attention mechanisms to enhance KV reuse. System-level approaches address memory management, scheduling, and hardware-aware designs to improve efficiency across diverse computing environments. Additionally, the survey provides an overview of both text and multimodal datasets and benchmarks used to evaluate these strategies. By presenting detailed taxonomies and comparative analyses, this work aims to offer useful insights for researchers and practitioners to support the development of efficient and scalable KV cache management techniques, contributing to the practical deployment of LLMs in real-world applications. The curated paper list for KV cache management is in: hrefhttps://github.com/TreeAI-Lab/Awesome-KV-Cache-Managementhttps://github.com/TreeAI-Lab/Awesome-KV-Cache-Management.

Key facts. Cited by 131 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 256 of its 328 references that are in the map. First posted on arXiv 27 Dec 2024.

Builds on

256 of 328 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  2. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning25 Oct
  3. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing24 Oct
  4. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression10 Aug

Other references · 252

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  10. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
38 more
  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  4. 2023Mistral 7Bcited by 3,164
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  12. 2024Mixtral of Expertscited by 1,823
  13. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  14. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  15. 2020Rethinking Attention with Performerscited by 1,601
  16. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  17. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  18. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  19. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  20. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  21. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  22. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  23. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  24. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  25. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  26. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  27. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  28. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  29. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  30. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  31. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  32. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  33. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  34. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  35. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  36. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  37. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  38. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744

50 most cited of 252 All 252 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 54; 2026: 77.

131 papers.

Sort:

By area:

  1. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 74
  2. 2025Oolong: Evaluating Long Context Reasoning and Aggregation Capabilitiescited by 40
  3. 2025Measuring the environmental impact of delivering AI at Google Scalecited by 32
  4. 2025d2Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Cachingcited by 25
  5. 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Modelscited by 24
  6. 2026TriAttention: Efficient Long Reasoning with Trigonometric KV Compressioncited by 21
  7. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inferencecited by 20
  8. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 19
  9. 2025Selective KV-Cache Sharing to Mitigate Timing Side-Channels in LLM Inferencecited by 18
  10. 2025xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extractioncited by 16
  11. 2025Comba: Improving Bilinear RNNs with Closed-loop Controlcited by 15
  12. 2025The Pitfalls of KV Cache Compressioncited by 15
38 more
  1. 2025Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Modelscited by 14
  2. 2025Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Querycited by 14
  3. 2025ParallelMuse: Agentic Parallel Thinking for Deep Information Seekingcited by 14
  4. 2026PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cachecited by 13
  5. 2025KV Cache Compression for Inference Efficiency in LLMs: A Reviewcited by 12
  6. 2025A Survey on Transformer Context Extension: Approaches and Evaluationcited by 11
  7. 2025Artificial Hippocampus Networks for Efficient Long-Context Modelingcited by 11
  8. 2026Baichuan-M3: Modeling Clinical Inquiry for Reliable Medical Decision-Makingcited by 11
  9. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Evictioncited by 11
  10. 2025Quantize What Counts: More for Keys, Less for Valuescited by 10
  11. 2025CommonKV: Compressing KV Cache with Cross-layer Parameter Sharingcited by 9
  12. 2026ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothingcited by 9
  13. 2026OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantizationcited by 9
  14. 2026FASA: Frequency-aware Sparse Attentioncited by 8
  15. 2025AttentionPredictor: Temporal Patterns Matter for KV Cache Compressioncited by 7
  16. 2025Homogeneous Keys, Heterogeneous Values: Exploiting Local KV Cache Asymmetry for Long-Context LLMscited by 7
  17. 2025Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modelingcited by 7
  18. 2025PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inferencecited by 7
  19. 2025Value-Guided KV Compression for LLMs via Approximated CUR Decompositioncited by 7
  20. 2025G-KV: Decoding-Time KV Cache Eviction with Global Attentioncited by 7
  21. 2026XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compressioncited by 7
  22. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMscited by 7
  23. 2025Survey and Experiments on Mental Disorder Detection via Social Media: From Large Language Models and RAG to Agentscited by 6
  24. 2025Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformerscited by 6
  25. 2026A Mechanistic View on Video Generation as World Models: State and Dynamicscited by 6
  26. 2026MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inferencecited by 6
  27. 2026IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inferencecited by 6
  28. 2025Compression Barriers for Autoregressive Transformerscited by 5
  29. 2025LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation frameworkcited by 5
  30. 2026SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Servingcited by 5
  31. 2026Rethinking KV Cache Eviction via a Unified Information-Theoretic Objectivecited by 5
  32. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inferencecited by 5
  33. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyondcited by 5
  34. 2026A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Designcited by 5
  35. 2025Cost-Optimal Grouped-Query Attention for Long-Context Modelingcited by 4
  36. 2025OSCAR: Online Soft Compression And Rerankingcited by 4
  37. 2025Comet: Accelerating Private Inference for Large Language Model by Predicting Activation Sparsitycited by 4
  38. 2025KV-Efficient VLA: A Method to Speed up Vision Language Models with RNN-Gated Chunked KV Cachecited by 4

50 most cited of 131 All 131 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  2. 2026IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs
  3. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  4. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
  5. 2025WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference
  6. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  7. 2024CORM: Cache Optimization with Recent Message for Large Language Model Inference
  8. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference