Adaptive KV-Cache Compression without Manually Setting Budget
On the map: Efficient LLM inference and serving → KV cache optimization
Abstract (arXiv)
Large language models (LLMs) inference relies heavily on KV-caches to accelerate autoregressive decoding, but the resulting memory footprint grows rapidly with sequence length, posing significant efficiency challenges. Current KV-cache compression methods suffer from a Procrustes' bed problem: they force diverse workloads into fixed compression ratios, leading to suboptimal resource allocation and inference performance. To this end, we present GVote, an adaptive KV-cache compression scheme that eliminates manual budget specification while achieving superior accuracy-efficiency trade-offs. GVote operates on the principle that the important keys are the aggregation of keys required by future queries. The method predicts future query attention demands by Monte-Carlo style sampling potential queries and aggregating selected keys to determine the optimal cache budget without manual specification. Experimental evaluation demonstrates GVote's effectiveness across multiple benchmarks, including GSM8K, RULER and Longbench. Compared to baselines, GVote exhibits 2× memory reduction while the accuracy maintains higher or comparable.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 18 of its 21 references that are in the map. First posted on arXiv 3 Sep 2025.
Builds on
18 of 21 references.
Closest ancestors · 6
Same direction, newest first
- 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences16 Mar
- 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption25 Jul
- 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
- 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
- 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
Other references · 12
- 2017Attention Is All You Needcited by 59,932
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2020Longformer: The Long-Document Transformercited by 3,362
- 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2024Qwen2 Technical Reportcited by 2,076
- 2020Big Bird: Transformers for Longer Sequencescited by 1,673
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
- 2019AutoML: A Survey of the State-of-the-Artcited by 388
- 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
- 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM
- 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
- 2025H1B-KV: Hybrid One-Bit Caches for Memory-Efficient Large Language Model Inference
- 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
- 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
- 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
- 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference