# KV Cache Optimization Strategies for Scalable and Efficient LLM Inference

arXiv: https://arxiv.org/abs/2603.20397 · HTML: https://mlsanity.com/paper/2603.20397/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 20 Mar 2026.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023)
- [InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management](https://mlsanity.com/paper/2406.19707/) (2024)
- [Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference](https://mlsanity.com/paper/2407.11550/) (2024)
- [Kimi Linear: An Expressive, Efficient Attention Architecture](https://mlsanity.com/paper/2510.26692/) (2025)
- [ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference](https://mlsanity.com/paper/2410.21465/) (2024)
- [MiniCache: KV Cache Compression in Depth Dimension for Large Language Models](https://mlsanity.com/paper/2405.14366/) (2024)
- [KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction](https://mlsanity.com/paper/2505.23416/) (2025)
- [NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time](https://mlsanity.com/paper/2408.03675/) (2024)
- [RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression](https://mlsanity.com/paper/2502.14051/) (2025)
- [Log-Linear Attention](https://mlsanity.com/paper/2506.04761/) (2025)
- [InfiniPot: Infinite Context Processing on Memory-Constrained LLMs](https://mlsanity.com/paper/2410.01518/) (2024)
- [Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMs](https://mlsanity.com/paper/2503.00979/) (2025)
- [LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management](https://mlsanity.com/paper/2410.00428/) (2024)
- [KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation](https://mlsanity.com/paper/2411.17089/) (2024)
- [TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization](https://mlsanity.com/paper/2505.19586/) (2025)
- [HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashing](https://mlsanity.com/paper/2412.16187/) (2024)
- [4bit-Quantization in Vector-Embedding for RAG](https://mlsanity.com/paper/2501.10534/) (2025)
- [Qrazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring](https://mlsanity.com/paper/2501.13331/) (2025)
- [Local Linear Attention: An Optimal Interpolation of Linear and Softmax Attention For Test-Time Regression](https://mlsanity.com/paper/2510.01450/) (2025)
- [AI Benchmark Democratization and Carpentry](https://mlsanity.com/paper/2512.11588/) (2025)
- [LiteCache: A Query Similarity-Driven, GPU-Centric KVCache Subsystem for Efficient LLM Inference](https://mlsanity.com/paper/2511.14510/) (2025)

## Built on by (most cited first, 3 of 3)

- [Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion](https://mlsanity.com/paper/2605.16579/) (2026)
- [Optimizing AI Inference Across the Deployment Stack](https://mlsanity.com/paper/2609.10550/) (2026)
- [The KV Cache Is the New Memory Wall](https://mlsanity.com/paper/2609.30854/) (2026)

## Abstract (arXiv)

The key-value (KV) cache is a foundational optimization in Transformer-based large language models (LLMs), eliminating redundant recomputation of past token representations during autoregressive generation. However, its memory footprint scales linearly with context length, imposing critical bottlenecks on GPU memory capacity, memory bandwidth, and inference throughput as production LLMs push context windows from thousands to millions of tokens. Efficient KV cache management has thus become a first-order challenge for scalable LLM deployment. This paper provides a systematic review of recent KV cache optimization techniques, organizing them into five principal directions: cache eviction, cache compression, hybrid memory solutions, novel attention mechanisms, and combination strategies. For each category we analyze the underlying mechanisms, deployment trade-offs, and empirical performance across memory reduction, throughput, and model accuracy metrics. We further map techniques to seven practical deployment scenarios, including long-context single requests, high-throughput datacenter serving, edge devices, multi-turn conversations, and accuracy-critical reasoning, providing actionable guidance for practitioners selecting among competing approaches. Our analysis reveals that no single technique dominates across all settings; instead, the optimal strategy depends on context length, hardware constraints, and workload characteristics, pointing toward adaptive, multi-stage optimization pipelines as a promising direction for future research.