mlsanityData to 7 Oct 2026

From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving

arXiv 2607.02574 ↗First posted 30 Jun 2026cs.DC cs.AI cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The key-value (KV) cache has become a first-order memory object in LLM serving rather than a temporary per-request tensor. This survey classifies more than thirty KV-management systems and frameworks using four axes: locality, lifetime, ownership, and substrate. The axes reveal five architectural archetypes -- local-paged, disaggregated-pipeline, shared-store, memory-pool, and hybrid-tier. Once workload and hardware are fixed, ownership accounts for much of the remaining design variance among distributed systems. The survey also audits current evaluations and identifies seven missing KV-specific measurements, linking them to open problems in fault tolerance, isolation, tiered eviction, speculative decoding, MoE serving, and shared-cache semantics.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 117 of its 146 references that are in the map. First posted on arXiv 30 Jun 2026.

Builds on

117 of 146 references.

Closest ancestors · 28

Same direction, newest first

  1. 2025TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale20 Dec
  2. 2025CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM Serving11 Dec
  3. 2025KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity5 Dec
  4. 2025Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management25 Nov
  5. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  6. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review8 Aug
  7. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneity24 Mar
  8. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference1 Feb
  9. 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
  10. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  11. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing24 Oct
  12. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
16 more
  1. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  2. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks11 Jul
  3. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  4. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  5. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  6. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  7. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  9. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget7 Apr
  10. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  11. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  12. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference14 Feb
  13. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  14. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  15. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  16. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 89

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
38 more
  1. 2017Billion-scale similarity search with GPUscited by 2,477
  2. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  6. 2024Mixtral of Expertscited by 1,785
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2020Reformer: The Efficient Transformercited by 1,658
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  13. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  14. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  15. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  16. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  17. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  18. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  19. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  20. 2020Efficient Transformers: A Surveycited by 784
  21. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  22. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  23. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  24. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  25. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  26. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  27. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  28. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  29. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  30. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  31. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  32. 2022Efficiently Scaling Transformer Inferencecited by 529
  33. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  34. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  35. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  36. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  37. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  38. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359

50 most cited of 89 All 89 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
  2. 2026DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memory
  3. 2026Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management
  4. 2026Unified KV Pooling to Accelerate Long-Context LLM Serving
  5. 2026Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
  6. 2026ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse
  7. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration
  8. 2025AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving