mlsanityData to 14 Oct 2026

The KV Cache Is the New Memory Wall

arXiv 2609.30854 ↗First posted 25 Sep 2026cs.DC cs.LG cs.PFcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization also fits: Serving systems and request scheduling

Abstract (arXiv)

Autoregressive LLM inference at long context is bounded by memory bandwidth, not arithmetic throughput, and the binding resource shifts from model weights to the Key-Value (KV) cache as sequence length grows. For Llama-3-70B in BF16, the 140 GB weight footprint exceeds the 80 GB HBM of a single accelerator, and one 128k-token sequence adds 42 GB of KV cache. Techniques that compress, evict, page, share, or offload KV state have proliferated, but reported gains use inconsistent workloads, hardware, and quality metrics, preventing cross-paper comparison. This SoK paper unifies the field analytically, with a protocol that strictly separates derived and reported claims. We derive closed-form arithmetic intensity as a decaying function of context length, parameterized by hardware topology for NVIDIA H100, NVIDIA B200, and AMD MI300X, including per-die bandwidth partitioning and the crossover lengths where KV traffic overtakes weight traffic. We classify the literature into five domains, quantization, token eviction, KV paging, prefix caching, and heterogeneous tiering, evaluating one method per domain at 128k context under a single protocol. The central finding is a three-regime structure: below a hardware-specific crossover, weight traffic dominates and KV compression yields negligible speedup; beyond it, KV traffic dominates and each domain trades quality for bandwidth savings approaching the roofline bound. Paging and prefix sharing are lossless but address capacity, not bandwidth. Quantization and eviction cut bandwidth directly, with degradation that accelerates below 4-bit precision and turns discontinuous for eviction on position-sensitive tasks. Tiering converts the bandwidth wall into an interconnect problem bounded by PCIe or NVLink rather than HBM. We close with design rules for selecting a compression domain given hardware, context length, and quality budget.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 30 of its 44 references that are in the map. First posted on arXiv 25 Sep 2026.

Builds on

30 of 44 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference20 Mar
  2. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  3. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  4. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  5. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  7. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  8. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  9. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  11. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2024Mixtral of Expertscited by 1,823
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  11. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  12. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
7 more
  1. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  2. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  3. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  4. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  5. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  6. 2025AI Benchmark Democratization and Carpentrycited by 3
  7. 2026Optimizing AI Inference Across the Deployment Stackcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
  2. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  3. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  4. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  5. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving
  6. 2026CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM
  7. 2025Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits
  8. 2025Accelerating LLM Inference Throughput via Asynchronous KV Cache Prefetching