mlsanityData to 7 Oct 2026

SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer

arXiv 2607.28150 ↗First posted 30 Jul 2026cs.DCcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Disaggregating the prefill and decoding stages of large language model (LLM) inference into two separate sets of nodes is widely adopted in today's LLM serving systems. However, such an architecture poses significant challenges for self-hosted LLM deployments on rented cloud instances, since transferring enormous key-value (KV) caches between disaggregated nodes can easily saturate the limited inter-node network bandwidth. In this paper, we propose to mitigate the network bottleneck by selectively transferring essential KV cache entries across the two stages. There are two challenges to achieve selective KV cache transfer, i.e., accurate KV selection during the prefill stage, and efficient KV fetching during the decoding stage. To address these challenges, we design SmartGen, a KV cache transfer engine that allows seamless disaggregated LLM inference with three data transfer paths. Specifically, we leverage 1) a profile-based proactive transfer path to identify and push essential KV cache entries to the decoding node during the prefill stage, 2) a parallel on-demand transfer path to simultaneously fetch remote and local KV cache entries during the decoding stage, and 3) a speculative transfer path to finally deliver all KV caches to the decoding node. Experimental results show that SmartGen reduces time-to-second-token by up to 4.3x compared with the typical full KV cache transfer approach while offering comparable subsequent decoding performance and accuracy.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 61 of its 86 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

61 of 86 references.

Closest ancestors · 19

Same direction, newest first

  1. 2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation19 Jan
  2. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider3 Jun
  3. 2025HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inference3 Jun
  4. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference5 May
  5. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneity24 Mar
  6. 2025HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference5 Feb
  7. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving2 Jan
  8. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  9. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  10. 2024Fast State Restoration in LLM Serving with HCache7 Oct
  11. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  12. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
7 more
  1. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  2. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  3. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving4 Mar
  4. 2023Stateful Large Language Model Serving with Pensieve9 Dec
  5. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2025Gemma 3 Technical Reportcited by 1,686
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  10. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  11. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  12. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
30 more
  1. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  2. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  3. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  5. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  6. 2021Efficient Attentions for Long Document Summarizationcited by 342
  7. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  8. 2025Phi-4-reasoning Technical Reportcited by 146
  9. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  10. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110
  11. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
  12. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  13. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 85
  14. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 83
  15. 2023Fairness in Serving Large Language Modelscited by 77
  16. 2023Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Methodcited by 76
  17. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
  18. 2020Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloadscited by 56
  19. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 54
  20. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  21. 2024BLITZSCALE: Fast and Live Large Model Autoscaling with O(1) Host Cachingcited by 23
  22. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  23. 2025KVComm: Enabling Efficient LLM Communication through Selective KV Sharingcited by 16
  24. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domainscited by 14
  25. 2025IC-Cache: Efficient Large Language Model Serving via In-context Cachingcited by 13
  26. 2025WaferLLM: Large Language Model Inference at Wafer Scalecited by 13
  27. 2025Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantizationcited by 13
  28. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Schedulingcited by 10
  29. 2024PhoenixOS: Concurrent OS-level GPU Checkpoint and Restore with Validated Speculationcited by 6
  30. 2025Spark Transformer: Reactivating Sparsity in FFN and Attentioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026CacheBridge: Efficient Cross-Model KV Cache Transfercited by 2
  2. 2026Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Managementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  2. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  3. 2024A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference
  4. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation
  5. 2026PDD: Unleashing Economical and Flexible Heterogeneous LLM Inference via Cross-Datacenter Prefill-Decode Disaggregation
  6. 2026Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction
  7. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
  8. 2024KVDirect: Distributed Disaggregated LLM Inference