BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
On the map: Efficient LLM inference and serving → KV cache optimization
Abstract (arXiv)
In Large Language Model (LLM) inference, Key-Value (KV) caches (KV-caches) are essential for reducing time complexity. However, they result in a linear increase in GPU memory as the context length grows. While recent work explores KV-cache eviction and compression policies to reduce memory usage, they often consider uniform KV-caches across all attention heads, leading to suboptimal performance. We introduce BaKlaVa, a method to allocate optimal memory for individual KV-caches across the model by estimating the importance of each KV-cache. Our empirical analysis demonstrates that not all KV-caches are equally critical for LLM performance. Using a one-time profiling approach, BaKlaVa assigns optimal memory budgets to each KV-cache. We evaluated our method on LLaMA-3-8B, and Qwen2.5-7B models, achieving up to a 70% compression ratio while keeping baseline performance and delivering up to an order-of-magnitude accuracy improvement at higher compression levels.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 26 references that are in the map. First posted on arXiv 18 Feb 2025.
Builds on
17 of 26 references.
Closest ancestors · 9
Same direction, newest first
- 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing24 Oct
- 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
- 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters18 Jun
- 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
- 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget7 Apr
- 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
- 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
- 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
Other references · 8
- 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
- 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
- 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
- 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
- 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 56
- 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
- 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
- 2024A Survey on Large Language Model Acceleration based on KV Cache Management
- 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference
- 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
- 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
- 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
- 2025WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference