mlsanityData to 14 Oct 2026

DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction

arXiv 2412.03131 ↗First posted 4 Dec 2024cs.LG cs.DCcited by 26

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language models (LLMs) demonstrate remarkable capabilities but face substantial serving costs due to their high memory demands, with the key-value (KV) cache being a primary bottleneck. State-of-the-art KV cache compression techniques, such as quantization and pruning, apply uniform treatment to both keys and values, and discard unimportant tokens entirely, overlooking the fine-grained distinctions in the significance of individual KV cache components. To address such limitations, we introduce textitDiffKV, a novel framework for efficient KV cache compression that exploits three levels of differentiation in the KV cache: (1) the differing impact of keys and values on attention computation, (2) the varying importance of tokens, and (3) the diverse dynamic sparsity patterns across attention heads. These levels of differentiation introduce irregular memory usage patterns across different requests and attention heads, posing significant scalability challenges for memory management. To address these challenges, DiffKV proposes an on-GPU memory manager that compacts fragmented free memory list into contiguous regions in parallel, effectively translating sparsity in the KV cache into performance gains. We evaluate DiffKV on several mainstream LLMs, including the emerging thinking models that generate extended chains of thought. DiffKV is able to compress the KV cache by 2.7× to 5.7× with near-lossless accuracy on complex workloads requiring sophisticated reasoning and long-generation capabilities, and enhances throughput by 1.9× to 5.4×. Source codes of DiffKV are available at https://github.com/zyqCSL/DiffKV.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 56 of its 81 references that are in the map. First posted on arXiv 4 Dec 2024.

Builds on

56 of 81 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  2. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  3. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  4. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  6. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  7. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  10. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  12. 2021Program Synthesis with Large Language Modelscited by 3,637
34 more
  1. 2016Pointer Sentinel Mixture Modelscited by 3,096
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2024Mixtral of Expertscited by 1,823
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  11. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  12. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  13. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  14. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  15. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  16. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  17. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  18. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  19. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  20. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  21. 2022Efficiently Scaling Transformer Inferencecited by 539
  22. 2021A White Paper on Neural Network Quantizationcited by 472
  23. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  24. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  25. 2024Capabilities of Gemini Models in Medicinecited by 237
  26. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  27. 2024Evaluation of OpenAI o1: Opportunities and Challenges of AGIcited by 127
  28. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
  29. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  30. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  31. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  32. 2021FLAT: An Optimized Dataflow for Mitigating Attention Bottleneckscited by 39
  33. 2024A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?cited by 33
  34. 2024MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in LLMscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 19.

26 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  2. 2025Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMscited by 20
  3. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  4. 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Servingcited by 7
  5. 2026Gated Differentiable Working Memory for Long-Context Language Modelingcited by 6
  6. 2025Compression Barriers for Autoregressive Transformerscited by 5
  7. 2025DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Accelerationcited by 4
  8. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 4
  9. 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inferencecited by 3
  10. 2025xGR: Efficient Generative Recommendation Serving at Scalecited by 2
  11. 2026Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Servingcited by 2
  12. 2026SwiftCache: Efficient LLM Serving for Multi-turn Conversations with Heterogeneous KV Cache Sharingcited by 2
14 more
  1. 2026HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compressioncited by 2
  2. 2026PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantizationcited by 2
  3. 2025CaliDrop: KV Cache Compression with Calibrationcited by 1
  4. 2026TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inferencecited by 1
  5. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restorationcited by 1
  6. 2026MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compressioncited by 1
  7. 2026vToken: Token-Level Virtualization for Reclaimable KV Cachescited by 1
  8. 2026TokenStack: A Heterogeneous HBM-PIM Architecture and Runtime for Efficient LLM Inferencecited by 0
  9. 2026SPEAR: A System for Post-Quantization Error-Adaptive Recovery Enabling Efficient Low-Bit LLM Servingcited by 0
  10. 2026Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attentioncited by 0
  11. 2026HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Modelscited by 0
  12. 2026UniCache: Task- and Type-Aware KV Cache Compression for Unified Multimodal Modelscited by 0
  13. 2026Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Servingcited by 0
  14. 2026E8 Lattice Key Quantization for Low-Bit KV-Cache Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
  2. 2026EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction
  3. 2026HybridKV: Hybrid KV Cache Compression for Efficient Multimodal Large Language Model Inference
  4. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing
  5. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques
  6. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
  7. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  8. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption