mlsanityData to 7 Oct 2026

Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference

arXiv 2407.11550 ↗First posted 16 Jul 2024cs.CL cs.AIcited by 230

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large Language Models have excelled in various domains but face efficiency challenges due to the growing Key-Value (KV) cache required for long-sequence inference. Recent efforts aim to reduce KV cache size by evicting vast non-critical cache elements during runtime while preserving generation quality. However, these methods typically allocate compression budgets uniformly across all attention heads, ignoring the unique attention patterns of each head. In this paper, we establish a theoretical loss upper bound between pre- and post-eviction attention output, explaining the optimization target of prior cache eviction methods, while guiding the optimization of adaptive budget allocation. Base on this, we propose it Ada-KV, the first head-wise adaptive budget allocation strategy. It offers plug-and-play benefits, enabling seamless integration with prior cache eviction methods. Extensive evaluations on 13 datasets from Ruler and 16 datasets from LongBench, all conducted under both question-aware and question-agnostic scenarios, demonstrate substantial quality improvements over existing methods. Our code is available at https://github.com/FFY0/AdaKV.

Key facts. Cited by 230 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 46 of its 71 references that are in the map. First posted on arXiv 16 Jul 2024.

Builds on

46 of 71 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  2. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  3. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  5. 2024On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference9 Feb
  6. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  9. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 37

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2023Mistral 7Bcited by 3,065
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
25 more
  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  2. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  4. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  5. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  7. 2019Are Sixteen Heads Really Better than One?cited by 803
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  9. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  10. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  11. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  12. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  13. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  14. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  15. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  16. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  17. 2021Efficient Attentions for Long Document Summarizationcited by 342
  18. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  19. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 188
  20. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  21. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  22. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110
  23. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  24. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  25. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 67; 2026: 154.

230 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  2. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 96
  3. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  4. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  5. 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Modelscited by 52
  6. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  7. 2025AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understandingcited by 45
  8. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compressioncited by 41
  9. 2025Accelerating Streaming Video Large Language Models via Hierarchical Token Compressioncited by 34
  10. 2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspectivecited by 28
  11. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  12. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMscited by 24
38 more
  1. 2025d2Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Cachingcited by 24
  2. 2025Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compressioncited by 22
  3. 2025SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMscited by 21
  4. 2025ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Modelscited by 21
  5. 2024SCOPE: Optimizing Key-Value Cache Compression in Long-context Generationcited by 20
  6. 2025Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMscited by 20
  7. 2025SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruningcited by 19
  8. 2026TriAttention: Efficient Long Reasoning with Trigonometric KV Compressioncited by 18
  9. 2025LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoningcited by 17
  10. 2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compressioncited by 16
  11. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scorescited by 16
  12. 2025LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocationcited by 16
  13. 2024A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compressioncited by 15
  14. 2025Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modelingcited by 15
  15. 2025Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Querycited by 15
  16. 2025Pie: A Programmable Serving System for Emerging LLM Applicationscited by 15
  17. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inferencecited by 14
  18. 2025Taming the Fragility of KV Cache Eviction in LLM Inferencecited by 14
  19. 2026KVzap: Fast, Adaptive, and Faithful KV Cache Pruningcited by 14
  20. 2026Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Evictioncited by 14
  21. 2025FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Accelerationcited by 13
  22. 2025CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Evictioncited by 13
  23. 2025Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Modelscited by 13
  24. 2026Past- and Future-Informed KV Cache Policy with Salience Estimation in Autoregressive Video Diffusioncited by 13
  25. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Headcited by 12
  26. 2026LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generationcited by 11
  27. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compressioncited by 10
  28. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimizationcited by 10
  29. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Evictioncited by 10
  30. 2025Artificial Hippocampus Networks for Efficient Long-Context Modelingcited by 9
  31. 2025OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inferencecited by 9
  32. 2026ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothingcited by 9
  33. 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inferencecited by 8
  34. 2025CentroidKV: Efficient Long-Context LLM Inference via KV Cache Clusteringcited by 8
  35. 2025UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compressioncited by 8
  36. 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inferencecited by 8
  37. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 8
  38. 2024CritiPrefill: A Segment-wise Criticality-based Approach for Prefilling Acceleration in LLMscited by 7

50 most cited of 230 All 230 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
  2. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  3. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences
  4. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving
  5. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  6. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference
  7. 2026MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference
  8. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time