mlsanityData to 14 Oct 2026

FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling

arXiv 2504.03775 ↗First posted 3 Apr 2025cs.DC cs.AI cs.CLcited by 10

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Disaggregated inference has become an essential framework that separates the prefill (P) and decode (D) stages in large language model inference to improve throughput. However, the KV cache transfer faces significant delays between prefill and decode nodes. The block-wise calling method and discontinuous KV cache memory allocation increase the number of calls to the transmission kernel. Additionally, existing frameworks often fix the roles of P and D nodes, leading to computational imbalances. In this paper, we propose FlowKV, a novel disaggregated inference framework, which reduces the average transmission latency of KV cache by 96%, from 0.944s to 0.053s, almost eliminating the transfer time relative to the total request latency by optimizing the KV cache transfer. FlowKV introduces the Load-Aware Scheduler for balanced request scheduling and flexible PD node allocation. This design maximizes hardware resource utilization, achieving peak system throughput across various scenarios, including normal, computational imbalance, and extreme overload conditions. Experimental results demonstrate that FlowKV significantly accelerates inference by 15.2%-48.9% on LongBench dataset compared to the baseline and supports applications with heterogeneous GPUs.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 19 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

17 of 19 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024KVDirect: Distributed Disaggregated LLM Inference13 Dec
  2. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool25 Jun
  3. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  5. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug

Other references · 12

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  6. 2024A Survey on Large Language Models for Code Generationcited by 658
  7. 2020A Survey on Knowledge Graphs: Representation, Acquisition and Applicationscited by 643
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  9. 2023Large Language Models for Generative Information Extraction: A Surveycited by 185
  10. 2024Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Workscited by 38
  11. 2024ToNER: Type-oriented Named Entity Recognition with Generative Language Modelcited by 8
  12. 2024Mitigating Out-of-Entity Errors in Named Entity Recognition: A Sentence-Level Strategycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  2. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocitycited by 12
  3. 2026Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patchingcited by 5
  4. 2025Disaggregated Prefill and Decoding Inference System for Large Language Model Serving on Multi-Vendor GPUscited by 3
  5. 2026SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Servingcited by 3
  6. 2025STAR: Decode-Phase Rescheduling for LLM Inferencecited by 1
  7. 2026Stage-Aware Communication Scheduling for Disaggregated LLM Servingcited by 1
  8. 2026NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inferencecited by 1
  9. 2026SpectrumKV: Per-Token Mixed-Precision KV Cache Transfer for Prefill-Decode Disaggregated LLM Servingcited by 1
  10. 2026Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Managementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference
  2. 2026SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference
  3. 2026SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer
  4. 2025DOPD: A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Serving
  5. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
  6. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation
  7. 2025SpecPipe: Accelerating Pipeline Parallelism-based LLM Inference with Speculative Decoding
  8. 2026Analytical Power-Aware Provisioning for Prefill-Decode Disaggregated AI Inference