# ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression arXiv: https://arxiv.org/abs/2412.03213 · HTML: https://mlsanity.com/paper/2412.03213/ ## Key facts - Cited by 65 papers in the map (data through 14 Oct 2026). - Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. - Builds on 19 of its 25 references that are in the map. - First posted on arXiv 4 Dec 2024. ## Placed in - LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/) - Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/) ## Builds on (most cited first, 19 of 19) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024) - [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022) - [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023) - [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023) - [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023) - [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024) - [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024) - [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023) - [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019) - [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024) - [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023) - [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023) - [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024) - [InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management](https://mlsanity.com/paper/2406.19707/) (2024) - [Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference](https://mlsanity.com/paper/2403.09054/) (2024) - [ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching](https://mlsanity.com/paper/2403.17312/) (2024) - [Inf-MLLM: Efficient Streaming Inference of Multimodal Large Language Models on a Single GPU](https://mlsanity.com/paper/2409.09086/) (2024) ## Built on by (most cited first, 30 of 65) - [Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention](https://mlsanity.com/paper/2502.11089/) (2025) - [A Survey on Large Language Model Acceleration based on KV Cache Management](https://mlsanity.com/paper/2412.19442/) (2024) - [Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs](https://mlsanity.com/paper/2502.12216/) (2025) - [Efficient Attention Mechanisms for Large Language Models: A Survey](https://mlsanity.com/paper/2507.19595/) (2025) - [SCOPE: Optimizing Key-Value Cache Compression in Long-context Generation](https://mlsanity.com/paper/2412.13649/) (2024) - [FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference](https://mlsanity.com/paper/2505.13109/) (2025) - [RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference](https://mlsanity.com/paper/2505.02922/) (2025) - [PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cache](https://mlsanity.com/paper/2601.04359/) (2026) - [KV Cache Steering for Controlling Frozen LLMs](https://mlsanity.com/paper/2507.08799/) (2025) - [SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Caching](https://mlsanity.com/paper/2504.00970/) (2025) - [Rectified Sparse Attention](https://mlsanity.com/paper/2506.04108/) (2025) - [Speed3R: Sparse Feed-forward 3D Reconstruction Models](https://mlsanity.com/paper/2603.08055/) (2026) - [HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading](https://mlsanity.com/paper/2502.12574/) (2025) - [ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference](https://mlsanity.com/paper/2602.08005/) (2026) - [CentroidKV: Efficient Long-Context LLM Inference via KV Cache Clustering](https://mlsanity.com/paper/2506.11418/) (2025) - [FASA: Frequency-aware Sparse Attention](https://mlsanity.com/paper/2602.03152/) (2026) - [A2ATS: Retrieval-Based KV Cache Reduction via Windowed Rotary Position Embedding and Query-Aware Vector Quantization](https://mlsanity.com/paper/2502.12665/) (2025) - [EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environments](https://mlsanity.com/paper/2509.17396/) (2025) - [Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attention](https://mlsanity.com/paper/2602.07223/) (2026) - [Trainable Dynamic Mask Sparse Attention](https://mlsanity.com/paper/2508.02124/) (2025) - [SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference](https://mlsanity.com/paper/2508.02751/) (2025) - [SpeContext: Enabling Efficient Long-context Reasoning with Speculative Context Sparsity in LLMs](https://mlsanity.com/paper/2512.00722/) (2025) - [Adaptive Computation Pruning for the Forgetting Transformer](https://mlsanity.com/paper/2504.06949/) (2025) - [Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Space](https://mlsanity.com/paper/2510.04476/) (2025) - [Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs](https://mlsanity.com/paper/2602.05191/) (2026) - [Self-Indexing KVCache: Predicting Sparse Attention from Compressed Keys](https://mlsanity.com/paper/2603.14224/) (2026) - [Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Models](https://mlsanity.com/paper/2606.08446/) (2026) - [Retrospective Sparse Attention for Efficient Long-Context Generation](https://mlsanity.com/paper/2508.09001/) (2025) - [LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequences](https://mlsanity.com/paper/2510.11292/) (2025) - [NOSA: Native and Offloadable Sparse Attention](https://mlsanity.com/paper/2510.13602/) (2025) ## Abstract (arXiv) Large Language Models (LLMs) have been widely deployed in a variety of applications, and the context length is rapidly increasing to handle tasks such as long-document QA and complex logical reasoning. However, long context poses significant challenges for inference efficiency, including high memory costs of key-value (KV) cache and increased latency due to extensive memory accesses. Recent works have proposed compressing KV cache to approximate computation, but these methods either evict tokens permanently, never recalling them for later inference, or recall previous tokens at the granularity of pages divided by textual positions. Both approaches degrade the model accuracy and output quality. To achieve efficient and accurate recallable KV cache compression, we introduce ClusterKV, which recalls tokens at the granularity of semantic clusters. We design and implement efficient algorithms and systems for clustering, selection, indexing and caching. Experiment results show that ClusterKV attains negligible accuracy loss across various tasks with 32k context lengths, using only a 1k to 2k KV cache budget, and achieves up to a 2× speedup in latency and a 2.5× improvement in decoding throughput. Compared to SoTA recallable KV compression methods, ClusterKV demonstrates higher model accuracy and output quality, while maintaining or exceeding inference efficiency. Our code is available at https://github.com/sjtu-zhao-lab/ClusterKV.