ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Serving
On the map: Efficient LLM inference and serving → Serving systems and request scheduling
Abstract (arXiv)
Current large language model (LLM) inference systems universally deploy ultra-large-scale models using a combination of Tensor Parallelism (TP) and Pipeline Parallelism (PP). However, existing systems treat the model parallelism topology as a static configuration that cannot be flexibly adjusted at runtime. This rigid design creates a fundamental contradiction with the dynamically changing inference workloads in real-world scenarios. State-of-the-art systems lack online reconfiguration capabilities and can only switch configurations by restarting the service, resulting in several minutes of service interruption, KV cache loss, and prohibitive recomputation overhead. To address this problem, this paper presents ReMP, a runtime model parallelism reconfiguration framework that supports low downtime. ReMP achieves dynamic adjustment through three key techniques: (1) decoupling the model parallelism topology from runtime state to avoid full service reconstruction; (2) designing a two-dimensional KV cache migration mechanism to preserve reusable cache states after TP/PP changes; and (3) implementing end-to-end online reconfiguration. Experiments demonstrate that ReMP can complete most topology switches within 1-7 seconds on models ranging from 7B to 70B parameters, achieving speedups of tens to over a hundred times compared to the restart approach. Moreover, ReMP significantly outperforms fixed configurations under dynamic workloads, delivering superior performance in terms of TTFT, TPOT, and output throughput.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 12 of its 22 references that are in the map. First posted on arXiv 17 Jun 2026.
Builds on
12 of 22 references.
Closest ancestors · 6
Same direction, newest first
- 2026PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Serving14 Apr
- 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
- 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention7 May
- 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
- 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov
Other references · 6
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
- 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
- 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
- 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads
- 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
- 2026AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving
- 2025Seesaw: High-throughput LLM Inference via Model Re-sharding
- 2025Efficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRA
- 2025AnchorTP: Resilient LLM Inference with State-Preserving Elastic Tensor Parallelism
- 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
- 2026SparPR: Runtime KV-Cache Recovery for Resource-Constrained LLM Inference on Consumer Hardware