mlsanityData to 7 Oct 2026

ProServe: Unified Multi-Priority Request Scheduling for LLM Serving

arXiv 2512.12928 ↗First posted 15 Dec 2025cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The widespread deployment of large language models (LLMs) for interactive applications necessitates serving systems that can handle thousands of concurrent requests with diverse Service Level Objective (SLO) requirements. A critical yet often overlooked dimension in this context is the inherent priority difference among clients; for instance, business-critical functions demand higher performance guarantees, as fulfilling such requests yields significantly greater business value. However, existing LLM serving schedulers fail to jointly optimize for both SLO attainment and client-level priorities. To bridge this gap, we first formalize multi-priority request scheduling as a service gain maximization problem, where satisfying latency requirements for requests of different priorities contributes varying gain. We propose ProServe, a unified two-tier scheduling framework designed to maximize overall service gain. At the engine layer, SlideBatching dynamically adapts batch formation under varying loads, employing a sliding boundary mechanism to balance latency and priority differentiation. Considering potential preemption, efficient block management adopts asynchronous offloading, pipelined reloading, and adaptive copy-budget control to overlap computation with host-device block transfers. At the service layer, GoRouting performs gain-oriented and capability-aware dispatching across distributed instances, proactively reserving capacity for future high-priority or long requests. Extensive evaluation on four open-source and one industrial dataset shows that ProServe outperforms state-of-the-art baselines, improving system gain by up to 35% and SLO attainment by up to 52%.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 39 of its 56 references that are in the map. First posted on arXiv 15 Dec 2025.

Builds on

39 of 56 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025xLLM Technical Report16 Oct
  2. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling3 Oct
  3. 2025HFX: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scaling21 Aug
  4. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving4 Aug
  5. 2025Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference29 May
  6. 2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture17 May
  7. 2025Echo: Efficient Co-Scheduling of Hybrid Online-Offline Tasks for Large Language Model Serving1 Mar
  8. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location15 Jan
  9. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  10. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  11. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  12. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services25 Apr
5 more
  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  2. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  4. 2023Fairness in Serving Large Language Models31 Dec
  5. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  9. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  10. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  11. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  12. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
10 more
  1. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  2. 2021FLAT: An Optimized Dataflow for Mitigating Attention Bottleneckscited by 38
  3. 2025Strata: Hierarchical Context Caching for Long Context Language Model Servingcited by 25
  4. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  5. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 17
  6. 2025HydraInfer: Hybrid Disaggregated Scheduling for Multimodal Large Language Model Servingcited by 9
  7. 2025SLO-Aware Scheduling for Large Language Model Inferencescited by 7
  8. 2025FairBatching: Fairness-Aware Batch Formation for LLM Inferencecited by 6
  9. 2025PolyServe: Efficient Multi-SLO Serving at Scalecited by 5
  10. 2025FineServe: Precision-Aware KV Slab and Two-Level Scheduling for Heterogeneous Precision LLM Servingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Serving
  2. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Information
  3. 2025Past-Future Scheduler for LLM Serving under SLA Guarantees
  4. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  5. 2025Ascendra: Dynamic Request Prioritization for Efficient LLM Serving
  6. 2024Queue management for slo-oriented large language model serving
  7. 2025Niyama : Breaking the Silos of LLM Inference Serving
  8. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration