mlsanityData to 14 Oct 2026

Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption

arXiv 2605.19593 ↗First posted 19 May 2026cs.AI cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Modern deployments of Large Language Models (LLMs) increasingly require serving multiple models with diverse architectures, sizes, and specialization on shared, heterogeneous hardware. This setting introduces new challenges for resource allocation, dispatching, and scheduling, particularly under GPU memory constraints where partial CPU-GPU offloading and preemption become necessary. While existing systems primarily optimize throughput for a single model, comparatively little work addresses multi-model scheduling under these conditions. In this paper, we present an empirical study of how different LLMs behave across hardware platforms, focusing on the performance implications of layer offloading and preemption. We show that offloading leads to strongly non-linear and model-dependent degradation in decode throughput, with smaller models exhibiting sharper sensitivity to reduced GPU residency. We further demonstrate that preemption incurs substantial overhead, largely dominated by model state reload rather than key-value cache transfer, and that this cost varies significantly across models and hardware platforms. Additionally, we highlight the role of sequence length and interconnect bandwidth in amplifying data movement and execution inefficiencies. Based on these findings, we identify a set of key features that future schedulers must consider, including model-specific offloading sensitivity, workload characteristics, and the cost structure of preemption and data transfer. These insights provide guidance for the design of next-generation LLM serving systems capable of efficiently managing heterogeneous, multi-model workloads with hybrid CPU-GPU execution.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 29 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

20 of 29 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024FastSwitch: Optimizing Context Switching Efficiency in Fairness-aware Large Language Model Serving27 Nov
  2. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference2 Nov
  3. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  4. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving2 Apr
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  7. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov
  8. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug
  9. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline22 May
  10. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 10

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  4. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  5. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  6. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  7. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  8. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 80
  9. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 71
  10. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Lightweight Scheduling Policies for Continuously Batched LLM Serving Across GPU Architectures
  2. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving
  3. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
  4. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration
  5. 2026Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use
  6. 2025LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems
  7. 2024ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving
  8. 2026BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization