CompKV: Compensation-Aware KV Selection for Long-Context LLM Inference
On the map: Efficient LLM inference and serving → KV cache optimization
Abstract (arXiv)
Despite their strong performance, large language models (LLMs) are bottlenecked by KV cache memory traffic during long-context inference. Sparse attention is widely used to accelerate LLM inference by computing exact attention over a selected subset of tokens. To recover the contribution of tokens excluded from exact attention, recent methods apply coarse-grained compensation to the omitted attention tail. However, existing methods typically select tokens based on attention mass and only then compensate for the unselected tokens. This decoupled design overlooks their interaction: selection should prioritize tokens that would leave the largest compensation error if omitted. To address this limitation, we introduce CompKV, the first compensation-aware sparse attention framework that divides tokens into blocks and explicitly optimizes selection for the downstream compensation mechanism. Our theoretical analysis shows that the residual left by block-level mean compensation is governed by both block attention mass and within-block logit variation. We approximate this residual using compact block-level statistics, yielding a deployable selection criterion. We further develop an efficient asynchronous implementation. Experiments on RULER and LongBench-Pro show that CompKV performs best among the evaluated sparse baselines while delivering up to a 6.85× self-attention speedup over full attention.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 33 of its 35 references that are in the map. First posted on arXiv 22 Sep 2026.
Builds on
33 of 35 references.
Closest ancestors · 16
Same direction, newest first
- 2026FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving20 Aug
- 2026ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression31 Jul
- 2026FASA: Frequency-aware Sparse Attention3 Feb
- 2025SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs20 Mar
- 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
- 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression10 Aug
- 2024Palu: Compressing KV-Cache with Low-Rank Projection30 Jul
- 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
- 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
- 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
- 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
- 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
4 more
- 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
- 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
Other references · 17
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
- 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
- 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
- 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
- 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
- 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
- 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 89
5 more
- 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 40
- 2026LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmarkcited by 19
- 2025Multipole Attention for Efficient Long Context Reasoningcited by 14
- 2025A Mathematical Theory of Top-k Sparse Attention via Total Variation Distancecited by 4
- 2026UNIQUE: Universal Top-k Sparse Attention for Training-free Inference and Sparsity-aware Trainingcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
- 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
- 2026SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference
- 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
- 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
- 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
- 2025Draft-based Approximate Inference for LLMs
- 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression