mlsanityData to 14 Oct 2026

FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters

arXiv 2510.11938 ↗First posted 13 Oct 2025cs.DCcited by 9

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Serving Large Language Models (LLMs) in production faces significant challenges from highly variable request patterns and severe resource fragmentation in serverless clusters. Current systems rely on static pipeline configurations that struggle to adapt to dynamic workload conditions, leading to substantial inefficiencies. We present FlexPipe, a novel system that dynamically reconfigures pipeline architectures during runtime to address these fundamental limitations. FlexPipe decomposes models into fine-grained stages and intelligently adjusts pipeline granularity based on real-time request pattern analysis, implementing three key innovations: fine-grained model partitioning with preserved computational graph constraints, inflight pipeline refactoring with consistent cache transitions, and topology-aware resource allocation that navigates GPU fragmentation. Comprehensive evaluation on an 82-GPU cluster demonstrates that FlexPipe achieves up to 8.5x better resource efficiency while maintaining 38.3% lower latency compared to state-of-the-art systems, reducing GPU reservation requirements from 75% to 30% of peak capacity.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 51 references that are in the map. First posted on arXiv 13 Oct 2025.

Builds on

24 of 51 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  2. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving2 Apr
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  8. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  9. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
  10. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  11. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 101
  12. 2020Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Providercited by 66
8 more
  1. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  2. 2023Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsitycited by 17
  3. 2022DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routingcited by 13
  4. 2021Automatic Graph Partitioning for Very Large-scale Deep Learningcited by 7
  5. 2021FaaSNet: Scalable and Fast Provisioning of Custom Serverless Container Runtimes at Alibaba Cloud Function Computecited by 7
  6. 2023SuperScaler: Supporting Flexible DNN Parallelization via a Unified Abstractioncited by 5
  7. 2022Groundhog: Efficient Request Isolation in FaaScited by 3
  8. 2024SeBS-Flow: Benchmarking Serverless Cloud Function Workflowscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Servingcited by 3
  2. 2026Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inferencecited by 2
  3. 2026Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUscited by 2
  4. 2026Coordinated Scheduling for MoE LLM Servingcited by 2
  5. 2026Multi-Layer Scheduling for MoE-Based LLM Reasoningcited by 1
  6. 2026A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variabilitycited by 0
  7. 2026Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUscited by 0
  8. 2026CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Servingcited by 0
  9. 2026Co-occurrence Patterns of LoRA Adapters in Production Diffusion Model Inference Servicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  2. 2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration
  3. 2026Towards Resiliency in Large Language Model Serving with KevlarFlow
  4. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration
  5. 2026OpScale: Operator-level Provisioning and Autoscaling for LLM Serving
  6. 2026LLM-Driven Intent-Based Privacy-Aware Orchestration Across the Cloud-Edge Continuum ↗
  7. 2026FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving
  8. 2025PipeBoost: Resilient Pipelined Architecture for Fast Serverless LLM Scaling