mlsanityData to 14 Oct 2026

GraniKV: Asymmetric Granularity KV-Cache Paging for Multi-Agent Systems with Long Shared Prefix

arXiv 2608.15584 ↗First posted 16 Aug 2026cs.LG cs.AIcited by 0

On the map: Efficient LLM inference and serving → Efficient reasoning and agent inference

Abstract (arXiv)

Production paged-serving engines apply uniform paging granularity to the KV cache, even though the two regions of a multi-agent workload have opposite storage requirements: a long shared prefix demands contiguity, while the per-request suffix demands fine-grained allocation. We present GraniKV, a KV-cache layer that allocates the shared prefix in a contiguous HOT pool and the suffix in a token-level COLD pool, combined with a per-step dispatcher which selects the appropriate backend among dual backends for each regime (compute-, memory-, or communication-bound). To the best of our knowledge, GraniKV is the first system to apply asymmetric paging granularity to the KV cache of a production paged-serving engine. At Lp=16,K shared tokens GraniKV reaches 2.16×, 1.98×, and 1.57× output-token throughput over the production baseline on Llama-3.1-8B/TP=1, Qwen-2.5-14B/TP=2, and Qwen-2.5-32B/TP=4. The gain decomposes: cascade attention integration contributes the majority at saturation; the asymmetric storage layer adds 1.05--1.15× end-to-end while being what makes the batched-GEMM prefix backend possible at all. Under heterogeneous multi-agent serving with distinct prompts of different lengths, the attribution inverts: GraniKV sustains 1.95× while batch-global cascade collapses to parity --- the storage layer alone carries the win in the regime that motivates the paper.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient reasoning and agent inference, one of 12 directions in Efficient LLM inference and serving. Builds on 14 of its 23 references that are in the map. First posted on arXiv 16 Aug 2026.

Builds on

14 of 23 references.

References · 14

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  8. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  9. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  10. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 82
  11. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 73
  12. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 70
2 more
  1. 2026Dive into Claude Code: The Design Space of Today's and Future AI Agent Systemscited by 38
  2. 2025From Attention to Disaggregation: Tracing the Evolution of LLM Inferencecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing
  2. 2026LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents
  3. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows
  4. 2026Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Serving
  5. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications
  6. 2026ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Management
  7. 2026Pancake: Hierarchical Memory System for Multi-Agent LLM Serving
  8. 2026Workload-Aware Caching for Multi-Agent Systems