Priority-Aware Preemptive Scheduling for Mixed-Priority Workloads in MoE Inference
On the map: Mixture-of-experts → Efficient MoE training and inference systems
Abstract (arXiv)
Large Language Models have revolutionized natural language processing, yet serving them efficiently in data centers remains challenging due to mixed workloads comprising latency-sensitive (LS) and best-effort (BE) jobs. Existing inference systems employ iteration-level first-come-first-served scheduling, causing head-of-line blocking when BE jobs delay LS jobs. We introduce QLLM, a novel inference system designed for Mixture of Experts (MoE) models, featuring a fine-grained, priority-aware preemptive scheduler. QLLM enables expert-level preemption, deferring BE job execution while minimizing LS time-to-first-token (TTFT). Our approach removes iteration-level scheduling constraints, enabling the scheduler to preempt jobs at any layer based on priority. Evaluations on an Nvidia A100 GPU show that QLLM significantly improves performance. It reduces LS TTFT by an average of 65.5× and meets the SLO at up to 7 requests/sec, whereas the baseline fails to do so under the tested workload. Additionally, it cuts LS turnaround time by up to 12.8× without impacting throughput. QLLM is modular, extensible, and seamlessly integrates with Hugging Face MoE models.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 16 of its 29 references that are in the map. First posted on arXiv 12 Mar 2025.
Builds on
16 of 29 references.
References · 16
- 2017Attention Is All You Needcited by 59,932
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
- 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
- 2020Recipes for building an open-domain chatbotcited by 593
- 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
- 2020Towards a Human-like Open-Domain Chatbotcited by 467
- 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
- 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
4 more
- 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
- 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
- 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
- 2024FastSwitch: Optimizing Context Switching Efficiency in Fairness-aware Large Language Model Servingcited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026Multi-Layer Scheduling for MoE-Based LLM Reasoningcited by 1
- 2026Multi-Layer Scheduling for MoE-Based LLM Reasoning25 Feb 2026
Similar, not cited
Close in wording, no citation either way
- 2026EWSJF: An Adaptive Scheduler with Hybrid Partitioning for Mixed-Workload LLM Inference
- 2025From Tokens to Layers: Redefining Stall-Free Scheduling for MoE Serving with Layered Prefill
- 2025MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints
- 2026ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefill
- 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference
- 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
- 2025APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs
- 2025DuoServe-MoE: Dual-Phase Expert Prefetch and Caching for LLM Inference QoS Assurance