mlsanityData to 14 Oct 2026

PipeBoost: Resilient Pipelined Architecture for Fast Serverless LLM Scaling

arXiv 2503.17707 ↗First posted 22 Mar 2025cs.DCcited by 2

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

This paper presents PipeBoost, a low-latency LLM serving system for multi-GPU (serverless) clusters, which can rapidly launch inference services in response to bursty requests without preemptively over-provisioning GPUs. Many LLM inference tasks rely on the same base model (e.g., LoRA). To leverage this, PipeBoost introduces fault-tolerant pipeline parallelism across both model loading and inference stages. This approach maximizes aggregate PCIe bandwidth and parallel computation across GPUs, enabling faster generation of the first token. PipeBoost also introduces recovery techniques that enable uninterrupted inference services by utilizing the shared advantages of multiple GPUs. Experimental results show that, compared to state-of-the-art low-latency LLM serving systems, PipeBoost reduces inference latency by 31% to 49.8%. For certain models (e.g., OPT-1.3B), PipeBoost achieves cold-start latencies in the range of a few hundred microseconds.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 57 references that are in the map. First posted on arXiv 22 Mar 2025.

Builds on

28 of 57 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  2. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  3. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  8. 2023Large Language Modelscited by 780
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  10. 2023The Falcon Series of Open Language Modelscited by 535
  11. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  12. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
11 more
  1. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  2. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 101
  3. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 75
  4. 2020Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Providercited by 66
  5. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  6. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  7. 2021Benchmarking, Analysis, and Optimization of Serverless Function Snapshotscited by 12
  8. 2022No Provisioned Concurrency: Fast RDMA-codesigned Remote Fork for Serverless Computingcited by 8
  9. 2020Faasm: Lightweight Isolation for Efficient Stateful Serverless Computingcited by 7
  10. 2021FaaSNet: Scalable and Fast Provisioning of Custom Serverless Container Runtimes at Alibaba Cloud Function Computecited by 7
  11. 2024ENOVA: Autoscaling towards Cost-effective and Stable Serverless LLM Servingcited by 6

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocitycited by 12
  2. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Servingcited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2026PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
  2. 2024PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation
  3. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  4. 2025LOIP:Collaborative Lossless LLM Inference Serving with Offloading-based Pipeline Parallelism on Edge Devices
  5. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  6. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  7. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment
  8. 2026SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchips