NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference
On the map: Efficient LLM inference and serving → Serving systems and request scheduling
Abstract (arXiv)
Disaggregated LLM inference forces the KV cache to traverse the datacenter network before decoding begins, so transfer time enters directly into the Time to First Token (TTFT) budget. Current schedulers route on compute load and prefix-cache locality alone, ignoring the topological distance and dynamic congestion between prefill and decode instances. We close this gap with a thin operator-to-scheduler interface, the network cost oracle, and we prove that ignoring the network term renders cache-aware-only scheduling arbitrarily suboptimal as context length grows. NetKV, the O(|D|) per-request greedy that consumes this oracle, has tier rankings that are provably robust to stale telemetry. On a 64-GPU four-tier fat-tree simulator driven by Mooncake traces, NetKV reduces mean TTFT by up to 21.2% over round-robin and 17.6% over a tuned cache+load-aware scheduler, lifts SLO attainment by up to 20.1 percentage points, and keeps the Time Between Tokens overhead below 0.5 ms in every condition tested, with no changes to the transport, inference engine, or hardware.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 28 references that are in the map. First posted on arXiv 2 Jun 2026.
Builds on
17 of 28 references.
Closest ancestors · 5
Same direction, newest first
- 2026GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving12 Feb
- 2025FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling3 Apr
- 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
- 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
Other references · 12
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
- 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
- 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
- 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 55
- 2023AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Servingcited by 31
- 2022TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobscited by 30
- 2023CASSINI: Network-Aware Job Scheduling in Machine Learning Clusterscited by 26
- 2025Cloud abstractions for AI workloadscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026Can I Buy Your KV Cache?cited by 0
- 2026Can I Buy Your KV Cache?11 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2026Beyond Prediction: Tail-Aware Scheduling for LLM Inference
- 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
- 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
- 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving
- 2026CALVO: Improve Serving Efficiency for LLM Inferences with Intense Network Demands
- 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
- 2026Requests of a Feather Must Flock Together: Batch Size vs. Prefix Homogeneity in LLM Inference
- 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference