mlsanityData to 7 Oct 2026

WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving

arXiv 2512.09472 ↗First posted 10 Dec 2025cs.DC cs.LGcited by 2

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Deploying multiple models within shared GPU clusters is a key strategy to improve resource efficiency in large language model (LLM) serving. Existing multi-LLM serving systems improve GPU utilization at the cost of degraded inference performance, particularly time-to-first-token (TTFT). We attribute this degradation to the lack of awareness regarding future workload characteristics. In contrast, recent analyses have shown the strong periodicity and long-term predictability of real-world LLM serving workloads. In this paper, we propose one-for-many GPU prewarming, which proactively loads parameters from multiple models onto GPUs based on workload forecasts. These prewarmed weights enable the system to promptly instantiate serving instances upon encountering request bursts. We design and implement WarmServe, a multi-LLM serving system incorporating three key techniques: (1) a model placement algorithm that optimizes prewarming decisions to minimize cross-model prewarming interference, (2) a KV cache reservation strategy that repurposes idle KV cache space on running GPUs for prewarming new models, and (3) an efficient GPU memory switching mechanism for tensor management. Evaluation on real-world datasets shows that WarmServe reduces tail TTFT by up to 50.8× compared to the state-of-the-art autoscaling-based system, while supporting up to 2.5× higher request throughput than the GPU-sharing system.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 33 of its 71 references that are in the map. First posted on arXiv 10 Dec 2025.

Builds on

33 of 71 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production15 May
  2. 2025Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning6 May
  3. 2025PipeBoost: Resilient Pipelined Architecture for Fast Serverless LLM Scaling22 Mar
  4. 2025HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds21 Feb
  5. 2024KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Serving24 Dec
  6. 2024Queue management for slo-oriented large language model serving5 Jun
  7. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention7 May
  8. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  9. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving2 Apr
  10. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  11. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache5 Jan
  12. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 21

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  7. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  8. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  9. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  10. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  11. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  12. 2020Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Providercited by 65
9 more
  1. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  2. 2023AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Servingcited by 31
  3. 2025DeepServe: Serverless Large Language Model Serving at Scalecited by 17
  4. 2023Torpor: GPU-Enabled Serverless Computing for Low-Latency, Resource-Efficient Inferencecited by 13
  5. 2023SuperServe: Fine-Grained Inference Serving for Unpredictable Workloadscited by 12
  6. 2022No Provisioned Concurrency: Fast RDMA-codesigned Remote Fork for Serverless Computingcited by 8
  7. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5
  8. 2024Enabling Elastic Model Serving with MultiWorldcited by 1
  9. 2025Tangram: Accelerating Serverless LLM Loading through GPU Memory Reuse and Affinitycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Projectcited by 1
  2. 2026Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Startcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  2. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
  3. 2026Characterizing CPU-Induced Slowdowns in Multi-GPU LLM Inference
  4. 2024ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving
  5. 2025LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems
  6. 2026Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving
  7. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains
  8. 2026Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures