mlsanityData to 14 Oct 2026

ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression

arXiv 2412.03213 ↗First posted 4 Dec 2024cs.LG cs.AI cs.PFcited by 65

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) have been widely deployed in a variety of applications, and the context length is rapidly increasing to handle tasks such as long-document QA and complex logical reasoning. However, long context poses significant challenges for inference efficiency, including high memory costs of key-value (KV) cache and increased latency due to extensive memory accesses. Recent works have proposed compressing KV cache to approximate computation, but these methods either evict tokens permanently, never recalling them for later inference, or recall previous tokens at the granularity of pages divided by textual positions. Both approaches degrade the model accuracy and output quality. To achieve efficient and accurate recallable KV cache compression, we introduce ClusterKV, which recalls tokens at the granularity of semantic clusters. We design and implement efficient algorithms and systems for clustering, selection, indexing and caching. Experiment results show that ClusterKV attains negligible accuracy loss across various tasks with 32k context lengths, using only a 1k to 2k KV cache budget, and achieves up to a 2× speedup in latency and a 2.5× improvement in decoding throughput. Compared to SoTA recallable KV compression methods, ClusterKV demonstrates higher model accuracy and output quality, while maintaining or exceeding inference efficiency. Our code is available at https://github.com/sjtu-zhao-lab/ClusterKV.

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 19 of its 25 references that are in the map. First posted on arXiv 4 Dec 2024.

Builds on

19 of 25 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  11. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  12. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
7 more
  1. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  2. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  3. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  5. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 126
  6. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  7. 2024Inf-MLLM: Efficient Streaming Inference of Multimodal Large Language Models on a Single GPUcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 28; 2026: 35.

65 papers.

Sort:

By area:

  1. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  2. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  3. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMscited by 24
  4. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 22
  5. 2024SCOPE: Optimizing Key-Value Cache Compression in Long-context Generationcited by 20
  6. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inferencecited by 14
  7. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  8. 2026PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cachecited by 13
  9. 2025KV Cache Steering for Controlling Frozen LLMscited by 11
  10. 2025SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Cachingcited by 10
  11. 2025Rectified Sparse Attentioncited by 10
  12. 2026Speed3R: Sparse Feed-forward 3D Reconstruction Modelscited by 10
38 more
  1. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloadingcited by 9
  2. 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inferencecited by 9
  3. 2025CentroidKV: Efficient Long-Context LLM Inference via KV Cache Clusteringcited by 8
  4. 2026FASA: Frequency-aware Sparse Attentioncited by 8
  5. 2025A2ATS: Retrieval-Based KV Cache Reduction via Windowed Rotary Position Embedding and Query-Aware Vector Quantizationcited by 7
  6. 2025EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environmentscited by 7
  7. 2026Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attentioncited by 7
  8. 2025Trainable Dynamic Mask Sparse Attentioncited by 6
  9. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inferencecited by 6
  10. 2025SpeContext: Enabling Efficient Long-context Reasoning with Speculative Context Sparsity in LLMscited by 6
  11. 2025Adaptive Computation Pruning for the Forgetting Transformercited by 5
  12. 2025Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Spacecited by 5
  13. 2026Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMscited by 5
  14. 2026Self-Indexing KVCache: Predicting Sparse Attention from Compressed Keyscited by 5
  15. 2026Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Modelscited by 5
  16. 2025Retrospective Sparse Attention for Efficient Long-Context Generationcited by 4
  17. 2025LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequencescited by 4
  18. 2025NOSA: Native and Offloadable Sparse Attentioncited by 4
  19. 2025Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limitscited by 4
  20. 2025DynaKV: Enabling Accurate and Efficient Long-Sequence LLM Decoding on Smartphonescited by 4
  21. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 4
  22. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 4
  23. 2026DensityKV: Density-Guided KV Cache Compression for Long Video Generationcited by 4
  24. 2025Sparse Attention Remapping with Clustering for Efficient LLM Decoding on PIMcited by 3
  25. 2025LLM-MemCluster: Empowering Large Language Models with Dynamic Memory for Text Clusteringcited by 3
  26. 2026Lil: Less is Less When Applying Post-Training Sparse-Attention Algorithms in Long-Decode Stagecited by 3
  27. 2026VMonarch: Efficient Video Diffusion Transformers with Structured Attentioncited by 3
  28. 2026MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inferencecited by 3
  29. 2026ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compressioncited by 3
  30. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  31. 2026VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inferencecited by 2
  32. 2026CSAttention: Centroid-Scoring Attention for Accelerating LLM Inferencecited by 2
  33. 2026AQPIM: Breaking the PIM Capacity Wall for LLMs with In-Memory Activation Quantizationcited by 2
  34. 2026AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inferencecited by 2
  35. 2026SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXLcited by 2
  36. 2026SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfercited by 2
  37. 2025Learn from the Past: Fast Sparse Indexing for Large Language Model Decodingcited by 1
  38. 2025Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategiescited by 1

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
  2. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  3. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques
  4. 2026CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
  5. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
  6. 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
  7. 2026MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference
  8. 2026PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression