mlsanityData to 7 Oct 2026

DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs

arXiv 2412.14838 ↗First posted 19 Dec 2024cs.CLcited by 42

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Efficient KV cache management in LLMs is crucial for long-context tasks like RAG and summarization. Existing KV cache compression methods enforce a fixed pattern, neglecting task-specific characteristics and reducing the retention of essential information. However, we observe distinct activation patterns across layers in various tasks, highlighting the need for adaptive strategies tailored to each task's unique demands. Based on this insight, we propose DynamicKV, a method that dynamically optimizes token retention by adjusting the number of tokens retained at each layer to adapt to the specific task. DynamicKV establishes global and per-layer maximum KV cache budgets, temporarily retaining the maximum budget for the current layer, and periodically updating the KV cache sizes of all preceding layers during inference. Our method retains only 1.7% of the KV cache size while achieving ~85% of the Full KV cache performance on LongBench. Notably, even under extreme compression (0.9%), DynamicKV surpasses state-of-the-art (SOTA) methods by 11% in the Needle-in-a-Haystack test using Mistral-7B-Instruct-v0.2. The code will be released.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 37 of its 51 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

37 of 51 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  2. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention21 May
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  4. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  5. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  7. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  11. 2024Qwen2 Technical Reportcited by 2,076
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
18 more
  1. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  2. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  3. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  4. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  5. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  6. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  7. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  8. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  9. 2021Efficient Attentions for Long Document Summarizationcited by 342
  10. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  11. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  12. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  13. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  14. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  15. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110
  16. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 95
  17. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  18. 2024mathbbUSCD: Improving Code Generation of LLMs by Uncertainty-Aware Selective Contrastive Decodingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 30.

42 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  2. 2025OmniSVG: A Unified Scalable Vector Graphics Generation Modelcited by 71
  3. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  4. 2025SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Cachingcited by 9
  5. 2026ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothingcited by 9
  6. 2025Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processingcited by 7
  7. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMscited by 7
  8. 2026MKA: Memory-Keyed Attention for Efficient Long-Context Reasoningcited by 6
  9. 2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generationcited by 5
  10. 2026When Less Latent Leads to Better Relay: Information-Preserving Compression for Latent Multi-Agent LLM Collaborationcited by 5
  11. 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizingcited by 4
  12. 2025Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Evictioncited by 3
30 more
  1. 2026Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inferencecited by 3
  2. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3
  3. 2026ReasonAlloc: Hierarchical Decoding-Time KV Cache Budget Allocation for Reasoning Modelscited by 3
  4. 2026MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inferencecited by 3
  5. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  6. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  7. 2025Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimizationcited by 1
  8. 2025CaliDrop: KV Cache Compression with Calibrationcited by 1
  9. 2025TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inferencecited by 1
  10. 2026CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilationcited by 1
  11. 2026AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attentioncited by 1
  12. 2026Transactional Attention: Semantic Sponsorship for KV-Cache Retentioncited by 1
  13. 2026Exploring a Layer-Wise Design Space for KV Cache Evictioncited by 1
  14. 2026Finetuning Vision-Language-Action Models Requires Fewer Layers Than You Thinkcited by 1
  15. 2026Epiphany-Aware KV Cache Eviction Without the Attention Matrixcited by 1
  16. 2026RippleKV: Cross-Layer KV Cache Allocation via Perturbation Propagationcited by 1
  17. 2026Monte Carlo Estimation for KV Cache Evictioncited by 1
  18. 2025KVCrush: Key value cache size-reduction using similarity in head-behaviourcited by 0
  19. 2026Slow-Fast Inference: Training-Free Inference Acceleration via Within-Sentence Support Stabilitycited by 0
  20. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategiescited by 0
  21. 2026HieraSparse: Hierarchical Semi-Structured Sparse KV Attentioncited by 0
  22. 2026AnchorKV: Safety-Aware KV Cache Compression via Soft Penalty with a Refusal Anchorcited by 0
  23. 2026HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decodingcited by 0
  24. 2026Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attentioncited by 0
  25. 2026Parser States Already Know: Structure-Conditioned KV Persistence for Structured Generationcited by 0
  26. 2026ECOKV: Geometry-Aware KV Cache Eviction via Complementary Diversity Metricscited by 0
  27. 2026What Must Survive? Exact Task-Information--State Frontiers Under Partial Task Revelationcited by 0
  28. 2026Risk-Controlled KV-Cache Eviction: From Memory Budgets to Risk Targetscited by 0
  29. 2026CORE: COverage CAlibration and Evicted-Mass REdistribution for KV Cachecited by 0
  30. 2026WakeKV: Reactive, Reversible KV Residency for Heads That Change Their Mindscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference
  2. 2025EvolKV: Evolutionary KV Cache Compression for LLM Inference
  3. 2025LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation
  4. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression
  5. 2026KV-Kaizen: Learning Context-Adaptive Cache Compression Choices
  6. 2025ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMs
  7. 2024ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty
  8. 2026KV Cache Offloading for Context-Intensive Tasks