mlsanityData to 14 Oct 2026

ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference

arXiv 2502.00299 ↗First posted 1 Feb 2025cs.CLcited by 60

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large Language Models (LLMs) require significant GPU memory when processing long texts, with the key value (KV) cache consuming up to 70% of total memory during inference. Although existing compression methods reduce memory by evaluating the importance of individual tokens, they overlook critical semantic relationships between tokens, resulting in fragmented context and degraded performance. We introduce ChunkKV, which fundamentally reimagines KV cache compression by treating semantic chunks - rather than isolated tokens - as basic compression units. This approach preserves complete linguistic structures and contextual integrity, ensuring that essential meaning is retained even under aggressive compression. Our innovation includes a novel layer-wise index reuse technique that exploits the higher cross-layer similarity of preserved indices in ChunkKV, reducing computational overhead and improving throughput by 26.5%. Comprehensive evaluations on challenging benchmarks: LongBench, Needle-In-A-HayStack, GSM8K, and JailbreakV demonstrate that ChunkKV outperforms state-of-the-art methods by up to 8.7% in precision while maintaining the same compression ratio. These results confirm that semantic-aware compression significantly enhances both efficiency and performance for long-context LLM inference, providing a simple yet effective solution to the memory bottleneck problem. The code is available at hrefhttps://github.com/NVIDIA/kvpresslink.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 72 of its 104 references that are in the map. First posted on arXiv 1 Feb 2025.

Builds on

72 of 104 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  2. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  3. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention21 May
  4. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  5. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Models17 May
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  7. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  9. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 63

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  10. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
38 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2024Qwen2 Technical Reportcited by 2,120
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  7. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  8. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  9. 2024Yi: Open Foundation Models by 01.AIcited by 719
  10. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  11. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  12. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  13. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  14. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  15. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  16. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  17. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  18. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  19. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  20. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  21. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 365
  22. 2023Adapting Language Models to Compress Contextscited by 356
  23. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  24. 2021Efficient Attentions for Long Document Summarizationcited by 345
  25. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
  26. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  27. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  28. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  29. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  30. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attackscited by 230
  31. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  32. 2024Many-Shot In-Context Learningcited by 206
  33. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  34. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  35. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  36. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 124
  37. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 112
  38. 2017DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applicationscited by 110

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 48.

60 papers.

Sort:

By area:

  1. 2025In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agentscited by 106
  2. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 47
  3. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 44
  4. 2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compressioncited by 16
  5. 2026Latent Context Compilation: Distilling Long Context into Compact Portable Memorycited by 13
  6. 2025Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attentioncited by 12
  7. 2025ScholarEval: Research Idea Evaluation Grounded in Literaturecited by 12
  8. 2025SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Cachingcited by 10
  9. 2026VCG-Bench: Towards A Unified Visual-Centric Benchmark for Structured Generation and Editingcited by 9
  10. 2025Value-Guided KV Compression for LLMs via Approximated CUR Decompositioncited by 7
  11. 2025FlowKV: Enhancing Multi-Turn Conversational Coherence in LLMs via Isolated Key-Value Cache Managementcited by 6
  12. 2026LongFlow: Efficient KV Cache Compression for Reasoning Modelscited by 6
38 more
  1. 2025SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Modelscited by 5
  2. 2026KEEP: A KV-Cache-Centric Memory Management System for Efficient Embodied Planningcited by 5
  3. 2026MDN: Parallelizing Stepwise Momentum for Delta Linear Attentioncited by 4
  4. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3
  5. 2026MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inferencecited by 3
  6. 2026RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Evictioncited by 3
  7. 2026ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agentscited by 3
  8. 2026CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilationcited by 2
  9. 2026ST-Lite: Training-Free KV Cache Compression with Spatio-Trajectory Guidance for Long-Horizon GUI Agentscited by 2
  10. 2026SPHERICAL KV: Angle-Domain Attention and Rate-Distortion Retention for Efficient Long-Context Inferencecited by 2
  11. 2026Epiphany-Aware KV Cache Eviction Without the Attention Matrixcited by 2
  12. 2026KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compressioncited by 2
  13. 2026Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compressioncited by 2
  14. 2026Every Cache Entry Earns Its Place: Global Allocation of Resolution and Coverage for KV Cache Compressioncited by 2
  15. 2025Neural Attention Searchcited by 1
  16. 2025Training-free Context-adaptive Attention for Efficient Long Context Modelingcited by 1
  17. 2026ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Managementcited by 1
  18. 2026S3-Attention:Attention-Aligned Endogenous Retrieval for Memory-Bounded Long-Context Inferencecited by 1
  19. 2026CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inferencecited by 1
  20. 2026Slow-Fast Inference: Training-Free Inference Acceleration via Within-Sentence Support Stabilitycited by 1
  21. 2026CodeComp: Structural KV Cache Compression for Agentic Codingcited by 1
  22. 2026Transactional Attention: Semantic Sponsorship for KV-Cache Retentioncited by 1
  23. 2026AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reusecited by 1
  24. 2026Training-Inference Consistent Segmented Execution for Long-Context LLMscited by 1
  25. 2026Adaptive Mass-Segmented KV Compression for Long-Context Reasoningcited by 1
  26. 2026DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compressioncited by 1
  27. 2026SeDeM: Selective Decompression of Hidden-State Memories for Long-Context Question Answeringcited by 1
  28. 2026KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Modelscited by 1
  29. 2025WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inferencecited by 0
  30. 2026DynSplit-KV: Dynamic Semantic Splitting for KVCache Compression in Efficient Long-Context LLM Inferencecited by 0
  31. 2026PRISM: Breaking the O(n) Memory Wall in Long-Context LLM Inference via O(1) Photonic Block Selectioncited by 0
  32. 2026Latent-Condensed Transformer for Efficient Long Context Modelingcited by 0
  33. 2026ART: Attention Run-time Termination for Efficient Large Language Model Decodingcited by 0
  34. 2026SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inferencecited by 0
  35. 2026From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Servingcited by 0
  36. 2026FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inferencecited by 0
  37. 2026SelKV: Selective KV Cache Merging with Per-Token Merge-or-Drop and Attention Compensationcited by 0
  38. 2026Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligencecited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
  2. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  3. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression
  4. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
  5. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
  6. 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
  7. 2026GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs
  8. 2026AnchorKV: Anchor-Residual KV Cache Compression