# FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters

arXiv: https://arxiv.org/abs/2510.11938 · HTML: https://mlsanity.com/paper/2510.11938/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 24 of its 51 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management](https://mlsanity.com/paper/2406.19707/) (2024)
- [CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://mlsanity.com/paper/2405.16444/) (2024)
- [Llumnix: Dynamic Scheduling for Large Language Model Serving](https://mlsanity.com/paper/2406.03243/) (2024)
- [Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads](https://mlsanity.com/paper/2401.11181/) (2024)
- [DAPPLE: A Pipelined Data Parallel Approach for Training Large Models](https://mlsanity.com/paper/2007.01045/) (2020)
- [Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider](https://mlsanity.com/paper/2003.03423/) (2020)
- [MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving](https://mlsanity.com/paper/2404.02015/) (2024)
- [Teola: Towards End-to-End Optimization of LLM-based Applications](https://mlsanity.com/paper/2407.00326/) (2024)
- [Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity](https://mlsanity.com/paper/2309.10285/) (2023)
- [DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing](https://mlsanity.com/paper/2212.03597/) (2022)
- [Automatic Graph Partitioning for Very Large-scale Deep Learning](https://mlsanity.com/paper/2103.16063/) (2021)
- [FaaSNet: Scalable and Fast Provisioning of Custom Serverless Container Runtimes at Alibaba Cloud Function Compute](https://mlsanity.com/paper/2105.11229/) (2021)
- [SuperScaler: Supporting Flexible DNN Parallelization via a Unified Abstraction](https://mlsanity.com/paper/2301.08984/) (2023)
- [Groundhog: Efficient Request Isolation in FaaS](https://mlsanity.com/paper/2205.11458/) (2022)
- [SeBS-Flow: Benchmarking Serverless Cloud Function Workflows](https://mlsanity.com/paper/2410.03480/) (2024)

## Built on by (most cited first, 9 of 9)

- [PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Serving](https://mlsanity.com/paper/2604.12171/) (2026)
- [Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inference](https://mlsanity.com/paper/2602.00328/) (2026)
- [Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs](https://mlsanity.com/paper/2605.04357/) (2026)
- [Coordinated Scheduling for MoE LLM Serving](https://mlsanity.com/paper/2606.15177/) (2026)
- [Multi-Layer Scheduling for MoE-Based LLM Reasoning](https://mlsanity.com/paper/2602.21626/) (2026)
- [A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variability](https://mlsanity.com/paper/2605.18750/) (2026)
- [Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUs](https://mlsanity.com/paper/2606.30391/) (2026)
- [CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving](https://mlsanity.com/paper/2607.04181/) (2026)
- [Co-occurrence Patterns of LoRA Adapters in Production Diffusion Model Inference Services](https://mlsanity.com/paper/2609.23321/) (2026)

## Abstract (arXiv)

Serving Large Language Models (LLMs) in production faces significant challenges from highly variable request patterns and severe resource fragmentation in serverless clusters. Current systems rely on static pipeline configurations that struggle to adapt to dynamic workload conditions, leading to substantial inefficiencies. We present FlexPipe, a novel system that dynamically reconfigures pipeline architectures during runtime to address these fundamental limitations. FlexPipe decomposes models into fine-grained stages and intelligently adjusts pipeline granularity based on real-time request pattern analysis, implementing three key innovations: fine-grained model partitioning with preserved computational graph constraints, inflight pipeline refactoring with consistent cache transitions, and topology-aware resource allocation that navigates GPU fragmentation. Comprehensive evaluation on an 82-GPU cluster demonstrates that FlexPipe achieves up to 8.5x better resource efficiency while maintaining 38.3% lower latency compared to state-of-the-art systems, reducing GPU reservation requirements from 75% to 30% of peak capacity.