# LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation

arXiv: https://arxiv.org/abs/2509.09754 · HTML: https://mlsanity.com/paper/2509.09754/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 11 Sep 2025.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling](https://mlsanity.com/paper/2406.02069/) (2024)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads](https://mlsanity.com/paper/2410.10819/) (2024)
- [Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference](https://mlsanity.com/paper/2407.11550/) (2024)
- [Retrieval Head Mechanistically Explains Long-Context Factuality](https://mlsanity.com/paper/2404.15574/) (2024)
- [Data Engineering for Scaling Language Models to 128K Context](https://mlsanity.com/paper/2402.10171/) (2024)
- [PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference](https://mlsanity.com/paper/2405.12532/) (2024)
- [LongCoder: A Long-Range Pre-trained Language Model for Code Completion](https://mlsanity.com/paper/2306.14893/) (2023)
- [Information Flow Routes: Automatically Interpreting Language Models at Scale](https://mlsanity.com/paper/2403.00824/) (2024)
- [Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters](https://mlsanity.com/paper/2406.12335/) (2024)
- [CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences](https://mlsanity.com/paper/2503.12491/) (2025)
- [CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective](https://mlsanity.com/paper/2502.03805/) (2025)
- [LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization](https://mlsanity.com/paper/2502.13922/) (2025)
- [Dodo: Dynamic Contextual Compression for Decoder-only LMs](https://mlsanity.com/paper/2310.02409/) (2023)
- [CORM: Cache Optimization with Recent Message for Large Language Model Inference](https://mlsanity.com/paper/2404.15949/) (2024)

## Built on by (most cited first, 16 of 16)

- [Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing](https://mlsanity.com/paper/2604.22782/) (2026)
- [OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMs](https://mlsanity.com/paper/2606.07577/) (2026)
- [Multi-Layered Memory Architectures for LLM Agents: An Experimental Evaluation of Long-Term Context Retention](https://mlsanity.com/paper/2603.29194/) (2026)
- [Novel Memory Forgetting Techniques for Autonomous AI Agents: Balancing Relevance and Efficiency](https://mlsanity.com/paper/2604.02280/) (2026)
- [Hybrid Associative Memories](https://mlsanity.com/paper/2603.22325/) (2026)
- [KV Cache Offloading for Context-Intensive Tasks](https://mlsanity.com/paper/2604.08426/) (2026)
- [Reformulating KV Cache Eviction Problem for Long-Context LLM Inference](https://mlsanity.com/paper/2605.07234/) (2026)
- [Do Value Vectors in Deep Layers Need Context from the Residual Stream?](https://mlsanity.com/paper/2606.02780/) (2026)
- [Exploring a Layer-Wise Design Space for KV Cache Eviction](https://mlsanity.com/paper/2606.15157/) (2026)
- [RippleKV: Cross-Layer KV Cache Allocation via Perturbation Propagation](https://mlsanity.com/paper/2608.08684/) (2026)
- [KV-Kaizen: Learning Context-Adaptive Cache Compression Choices](https://mlsanity.com/paper/2609.37988/) (2026)
- [Monte Carlo Estimation for KV Cache Eviction](https://mlsanity.com/paper/2610.07643/) (2026)
- [Whole-Piece Training for Symbolic Music Language Models via Full-Horizon Compressed Recurrence](https://mlsanity.com/paper/2602.19816/) (2026)
- [Developing Adaptive Context Compression Techniques for Large Language Models (LLMs) in Long-Running Interactions](https://mlsanity.com/paper/2603.29193/) (2026)
- [SelKV: Selective KV Cache Merging with Per-Token Merge-or-Drop and Attention Compensation](https://mlsanity.com/paper/2607.16213/) (2026)
- [WakeKV: Reactive, Reversible KV Residency for Heads That Change Their Minds](https://mlsanity.com/paper/2610.02713/) (2026)

## Abstract (arXiv)

KV Cache is commonly used to accelerate LLM inference with long contexts, yet its high memory demand drives the need for cache compression. Existing compression methods, however, are largely heuristic and lack dynamic budget allocation. To address this limitation, we introduce a unified framework for cache compression by minimizing information loss in Transformer residual streams. Building on it, we analyze the layer attention output loss and derive a new metric to compare cache entries across heads, enabling layer-wise compression with dynamic head budgets. Additionally, by contrasting cross-layer information, we also achieve dynamic layer budgets. LAVa is the first unified strategy for cache eviction and dynamic budget allocation that, unlike prior methods, does not rely on training or the combination of multiple strategies. Experiments with benchmarks (LongBench, Needle-In-A-Haystack, Ruler, and InfiniteBench) demonstrate its superiority. Moreover, our experiments reveal a new insight: dynamic layer budgets are crucial for generation tasks (e.g., code completion), while dynamic head budgets play a key role in extraction tasks (e.g., extractive QA). As a fully dynamic compression method, LAVa consistently maintains top performance across task types. Our code is available at https://github.com/MGDDestiny/Lava.