mlsanityData to 13 Oct 2026

Unified KV Pooling to Accelerate Long-Context LLM Serving

arXiv 2606.14779 ↗First posted 10 Jun 2026cs.ARcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long-context LLM serving requires offloading KV caches to host-memory and SSDs, but existing mechanisms are not designed for such long contexts. We observe significant inefficiencies in current KV caching in long contexts: high serving latency ~30.7 s, exceeding the typical TTFT requirement of 10 s by more than 3x. Our in-depth analysis explains two major reasons: (1) retrieval is serialized through host-memory and SSD, leaving other host-memory modules and SSDs underutilized, and (2) SSD-based KV retrieval spends 84% of its time in the kernel filesystem rather than actual device access. To address the problems, we propose unified KV pooling, which aggregates multiple host-memory modules and SSDs into a single logical pool and distributes KV caches across devices based on their bandwidth. To eliminate the filesystem overhead, we design KV-passthrough, which bypasses the kernel filesystem and directly accesses SSD-resident KV caches from user space via SPDK. Across evaluations on LLaMA 3.1-8B, GPT-OSS-20B, and Qwen3-30B-A3B, unified KV pooling reduces TTFT in long-contexts ~4.1x over state-of-the-art techniques, all making under 10 s. It also reduces blocked I/O time by up to 23.2x by eliminating filesystem overhead.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 32 references that are in the map. First posted on arXiv 10 Jun 2026.

Builds on

21 of 32 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  2. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving2 Jan
  3. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference8 Sep
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  5. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 15

  1. 2017Attention Is All You Needcited by 60,106
  2. 2024The Llama 3 Herd of Modelscited by 14,486
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,221
  4. 2025Qwen3 Technical Reportcited by 8,510
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,130
  7. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,945
  8. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,319
  9. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,180
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 572
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 570
  12. 2023Retrieval meets Long Context Large Language Modelscited by 115
3 more
  1. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 98
  2. 2024LongGenBench: Benchmarking Long-Form Generation in Long Context LLMscited by 46
  3. 2023Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Trainingcited by 31

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
  2. 2026Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
  3. 2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration
  4. 2026Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management
  5. 2026ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse
  6. 2026Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs
  7. 2026HBF Sucks? A Full-Stack Characterization of High-Bandwidth Flash for KV-Centric LLM Serving
  8. 2026TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash