MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference
On the map: Efficient LLM inference and serving → KV cache optimization
Abstract (arXiv)
Key--value (KV) cache compression is an effective way to reduce the memory overhead of large language model (LLM) inference, particularly for long-context workloads. However, existing compression methods make different trade-offs among accuracy, inference latency, and peak KV cache memory utilization, making a single fixed configuration unsuitable across different prompts and resource constraints. We introduce MetaKV, an adaptive framework that selects a KV cache compression configuration for each input prompt based on user-specified latency and peak memory budgets. MetaKV uses lightweight prediction models to estimate the end-to-end latency, peak memory, and probability of a correct response for each candidate configuration, and selects the configuration that best satisfies the latency-memory constraints while preserving accuracy. We evaluate MetaKV across ten configurations from three representative KV cache compression methods, KVQuant, H2O, and RocketKV, together with an uncompressed FP16 configuration, on four datasets covering mathematics, science, commonsense reasoning, and reading comprehension. Across a wide range of latency and peak memory constraints, MetaKV consistently outperforms the best static configuration, improving constrained success rate (CSR), the fraction of prompts answered correctly while satisfying both constraints, by approximately 0.07 on average and up to 0.135. These results demonstrate the benefit of adapting KV cache compression to individual prompts and latency-memory constraints. Code is available at https://github.com/MichaelWang0505/MetaKV.git
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 14 of its 17 references that are in the map. First posted on arXiv 7 Sep 2026.
Builds on
14 of 17 references.
Closest ancestors · 1
Same direction, newest first
Other references · 13
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
- 2023A Survey of Large Language Modelscited by 2,576
- 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
- 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
- 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
- 2025A Survey on Code Generation with LLM-based Agentscited by 106
- 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
- 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
1 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
- 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
- 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
- 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
- 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
- 2026CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
- 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression
- 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference