mlsanityData to 7 Oct 2026

FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving

arXiv 2602.22593 ↗First posted 26 Feb 2026cs.DCcited by 5

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Production LLM serving must simultaneously deliver high throughput, low latency, and sufficient context capacity under non-stationary traffic and mixed request requirements. Data parallelism (DP) maximizes throughput by running independent replicas, while tensor parallelism (TP) reduces per-request latency and pools memory for long-context inference. However, existing serving stacks typically commit to a static parallelism configuration at deployment; adapting to bursts, priorities, or long-context requests is often disruptive and slow. We present Flying Serving, a vLLM-based system that enables online DP-TP switching without restarting engine workers. Flying Serving makes reconfiguration practical by virtualizing the state that would otherwise force data movement: (i) a zero-copy Model Weights Manager that exposes TP shard views on demand, (ii) a KV Cache Adaptor that preserves request KV state across DP/TP layouts, (iii) an eagerly initialized Communicator Pool to amortize collective setup, and (iv) a deadlock-free scheduler that coordinates safe transitions under execution skew. Across three popular LLMs and realistic serving scenarios, Flying Serving improves performance by up to 4.79× under high load and 3.47× under low load while supporting latency- and memory-driven requests.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 51 references that are in the map. First posted on arXiv 26 Feb 2026.

Builds on

26 of 51 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity3 Dec
  2. 2025Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads20 Sep
  3. 2025Seesaw: High-throughput LLM Inference via Model Re-sharding9 Mar
  4. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  6. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  7. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug
  8. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  9. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  10. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  11. 2022Efficiently Scaling Transformer Inferencecited by 529
  12. 2019MLPerf Inference Benchmarkcited by 195
6 more
  1. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  2. 2023Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collectionscited by 15
  3. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Modelscited by 13
  4. 2025Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelismcited by 9
  5. 2025HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inferencecited by 3
  6. 2025Arctic Inference with Shift Parallelism: Fast and Efficient Open Source Inference System for Enterprise AIcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Startcited by 1
  2. 2026Moebius: Serving Mixture-of-Expert Models with Seamless Runtime Parallelism Switchcited by 1
  3. 2026FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Modelscited by 0
  4. 2026Crossflow: Prefill-Decode Elasticity for Agentic LLM Servingcited by 0
  5. 2026SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism
  2. 2026ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Serving
  3. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
  4. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving
  5. 2026CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving
  6. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughput
  7. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  8. 2026From Inference Engine to Inference Control Plane: Connecting vLLM, llm-d, and the Evolution of Efficient Distributed LLM Serving