Stateful KV Cache Management for LLMs: Balancing Space, Time, Accuracy, and Positional Fidelity
On the map: Efficient LLM inference and serving → KV cache optimization
Abstract (arXiv)
The Key-Value (KV) cache is integral to efficient autoregressive inference in large language models (LLMs), yet its unbounded growth in stateful multi-turn scenarios presents major challenges. This paper examines the interplay between KV cache management strategies, the architectural context limits of models like meta-llama/Meta-Llama-3-8b-instruct, and the often-overlooked integrity of positional encodings. Through empirical analysis using a stateful benchmarking framework, we show that LLM generation quality degrades sharply when the accumulated KV cache approaches or exceeds the model's trained context window (e.g., 8192 tokens for Llama 3), a failure mode distinct from GPU memory exhaustion. Common eviction strategies, even high-retention ones (e.g., 99% via AttentionTop), can worsen performance if they disrupt positional coherence. Because LLMs rely on consistent positional signals (e.g., RoPE), compacting a cache by removing non-contiguous tokens can scramble these signals and lead to degenerative outputs. We further show that simple strategies preserving contiguous context blocks (e.g., keeping an initial "gist") can yield more coherent generations than complex or positionally disruptive ones. We advocate for eviction techniques that respect architectural limits, preserve positional structure, and view "cache health" holistically beyond mere size.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 21 references that are in the map. First posted on arXiv 23 Oct 2025.
Builds on
17 of 21 references.
Closest ancestors · 2
Same direction, newest first
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
Other references · 15
- 2017Attention Is All You Needcited by 59,932
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2020Longformer: The Long-Document Transformercited by 3,362
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
- 2019Generating Long Sequences with Sparse Transformerscited by 1,523
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
- 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
- 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
- 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
3 more
- 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
- 2022Efficiently Scaling Transformer Inferencecited by 529
- 2023Effective Long-Context Scaling of Foundation Modelscited by 301
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
- 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
- 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
- 2024A Survey on Large Language Model Acceleration based on KV Cache Management
- 2026IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs
- 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
- 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
- 2026TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference