BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructure
On the map: Efficient LLM inference and serving → Serving systems and request scheduling
Abstract (arXiv)
Large language models (LLMs) are increasingly deployed in AI infrastructure, driving the need for high throughput, resource efficient serving systems. Disaggregated LLM serving, which separates prompt prefill from auto-regressive decode, has emerged as a promising architecture by isolating their heterogeneous compute and memory demands. However, current disaggregated systems face three key limitations: (i) static resource allocation cannot adapt to highly dynamic workloads, causing over-provisioning that wastes resources or under-provisioning that violates service level objectives (SLOs); (ii) inherent load imbalance between prefill and decode stages, where prefill is compute-bound and decode is memory-bound, causes under-utilization in one tier while the other becomes a bottleneck; and (iii) prefix cache aware routing skews load distribution, as high cache hit rate prefill nodes attract disproportionately more requests, further degrading balance and efficiency. To address these issues, we present BanaServe, a dynamic orchestration framework that continuously rebalances computational and memory resources across prefill and decode instances while eliminating hotspots induced by cache. BanaServe introduces layer level weight migration, attention level Key Value Cache (KV Cache) migration, and Global KV Cache Store sharing with layer wise overlapped transmission, enabling both coarse grained (layer level) and fine grained (attention level) load redistribution with minimal latency overhead. These mechanisms allow routers to perform purely load aware scheduling, unconstrained by cache placement. Compared to vLLM, BanaServe achieves 1.2x-3.9x higher throughput with 3.9%-78.4% lower total processing time, and outperforms DistServe by 1.1x-2.8x in throughput with 1.4%-70.1% latency reduction.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 41 references that are in the map. First posted on arXiv 15 Oct 2025.
Builds on
28 of 41 references.
Closest ancestors · 10
Same direction, newest first
- 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage28 Apr
- 2024KVDirect: Distributed Disaggregated LLM Inference13 Dec
- 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool25 Jun
- 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
- 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
- 2024A Survey on Efficient Inference for Large Language Models22 Apr
- 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
- 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
- 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov
Other references · 18
- 2017Attention Is All You Needcited by 60,361
- 2023GPT-4 Technical Reportcited by 18,396
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
- 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
- 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
- 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
6 more
- 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
- 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
- 2024BLITZSCALE: Fast and Live Large Model Autoscaling with O(1) Host Cachingcited by 25
- 2023Generative Artificial Intelligence for Software Engineering -- A Research Agendacited by 23
- 2023NoCodeGPT: A No-Code Interface for Building Web Apps with Language Modelscited by 4
- 2024StatuScale: Status-aware and Elastic Scaling Strategy for Microservice Applicationscited by 2
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2026PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving
- 2025Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scaling
- 2025PRESERVE: Prefetching Model Weights and KV-Cache in Distributed LLM Serving
- 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
- 2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture
- 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration
- 2025DOPD: A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Serving
- 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving