mlsanityData to 7 Oct 2026

PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Serving

arXiv 2604.12171 ↗First posted 14 Apr 2026cs.DC cs.LGcited by 3

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Pipeline parallelism (PP) is widely used to partition layers of large language models (LLMs) across GPUs, enabling scalable inference for large models. However, existing systems rely on static PP configurations that fail to adapt to dynamic settings, such as serverless platforms and heterogeneous GPU environments. Reconfiguring PP by stopping and redeploying service incurs prohibitive downtime, so reconfiguration must instead proceed live and in place, without interrupting inference. However, live in-place PP reconfiguration is fundamentally challenging. GPUs are already saturated with model weights and KV cache, leaving little room for new layer placements and necessitating KV cache resizing, at odds with systems like vLLM that preallocate for throughput. Moreover, maintaining KV consistency during execution is difficult: stop-and-copy introduces large pauses, while background synchronization risks inconsistency as states evolve. We present PipeLive, which enables live in-place PP reconfiguration with minimal disruption. PipeLive introduces a redesigned KV cache layout together with a co-designed extension to PageAttention, forming a unified mechanism for live KV resizing. It further adopts an incremental KV patching mechanism, inspired by live virtual machine migration, to synchronize KV states between source and target configurations and identify a safe switch point. PipeLive achieves a 2.5X reduction in time-to-first-token (TTFT) without KV cache overflow compared to disabling KV resizing. Furthermore, compared to a variant without KV patching, it reduces reconfiguration overhead from seconds to under 10ms, and improves TTFT and time-per-output-token (TPOT) by up to 54.7% and 14.7%, respectively.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 36 references that are in the map. First posted on arXiv 14 Apr 2026.

Builds on

23 of 36 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters13 Oct
  2. 2025HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds21 Feb
  3. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention7 May
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  5. 2024LLM Inference Unveiled: Survey and Roofline Model Insights26 Feb
  6. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  8. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov

Other references · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  10. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
  11. 2022Aryl: An Elastic Cluster Scheduler for Deep Learningcited by 27
  12. 2025Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithmscited by 16
3 more
  1. 2023Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collectionscited by 15
  2. 2025Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelismcited by 9
  3. 2025Towards Efficient and Practical GPU Multitasking in the Era of LLMcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Servingcited by 1
  2. 2026WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reusecited by 0
  3. 2026SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LLM-Driven Intent-Based Privacy-Aware Orchestration Across the Cloud-Edge Continuum ↗
  2. 2024PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation
  3. 2026PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
  4. 2026MOCAP: Wafer-Scale-Chip-Oriented Memory-Orchestrated Chunked Pipelining Framework for Prefill-Only LLM Inference
  5. 2026VPP: Virtual Pipeline Parallelism for Efficient Chunked Prefill in Long-Context LLM Inference
  6. 2025PipeBoost: Resilient Pipelined Architecture for Fast Serverless LLM Scaling
  7. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration
  8. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving