mlsanityData to 14 Oct 2026

Coordinated Scheduling for MoE LLM Serving

arXiv 2606.15177 ↗First posted 13 Jun 2026cs.DCcited by 2

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

Serving Mixture-of-Experts (MoE) large language models (LLMs) is challenging because dynamic request workloads interact with sparse expert routing, creating both data-parallel (DP) engine imbalance and expert-level hotspots. Existing LLM serving systems typically make these decisions in isolation: frontend schedulers route requests using coarse request counters, while backend expert balancers rely mainly on aggregate expert activation counts. This separation prevents the serving system from reacting to fine-grained engine pressure, backend MoE pressure, and source-dependent expert traffic. To address this gap, we propose Gimbal, a coordinated cross-level scheduling system for efficient MoE-based LLM serving. First, Gimbal presents a fine-grained DP-engine scheduler that uses online backend pressure signals, including key-value (KV) cache usage, remaining prefill work, queue pressure, and MoE expert pressure, to dispatch requests away from overloaded engines. Inside each engine, Gimbal further applies a lightweight prefill-aware queue ordering policy with aging to reduce head-of-line blocking without output-length prediction. Second, Gimbal extends expert load balancing with online source-DP-to-expert routing statistics and uses a heuristic guided by a mixed-integer nonlinear program (MINLP) to place experts while jointly considering expert load, source-aware communication, and migration stability. Our evaluation shows that Gimbal reduces average Time To First Token (TTFT) by 42.9% and average Time Per Output Token (TPOT) by 33.3% compared with the state-of-the-art serving system vLLM, while improving high-load request throughput by 3.0%.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 17 of its 53 references that are in the map. First posted on arXiv 13 Jun 2026.

Builds on

17 of 53 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inference15 Dec
  2. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing10 Feb

Other references · 15

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2024Mixtral of Expertscited by 1,823
  3. 2024DeepSeek-V3 Technical Reportcited by 811
  4. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  5. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  6. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 70
  7. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 66
  8. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Servingcited by 23
  9. 2024Efficiently Serving Large Multimodal Models Using EPD Disaggregationcited by 16
  10. 2025Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregationcited by 14
  11. 2025HydraInfer: Hybrid Disaggregated Scheduling for Multimodal Large Language Model Servingcited by 9
  12. 2025FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusterscited by 9
3 more
  1. 2026Not All Prefills Are Equal: PPD Disaggregation for Multi-turn LLM Servingcited by 6
  2. 2025SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inferencecited by 4
  3. 2025Efficient LLM Serving on Hybrid Real-time and Best-effort Requestscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026RoutePack: Expert Placement and Attention-Aware Data Packing for MoE Reinforcement Learningcited by 0
  2. 2026Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-Layer Scheduling for MoE-Based LLM Reasoning
  2. 2026MoEless: Efficient MoE LLM Serving via Serverless Computing
  3. 2026CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training
  4. 2025MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs
  5. 2025DuoServe-MoE: Dual-Phase Expert Prefetch and Caching for LLM Inference QoS Assurance
  6. 2024Ada-K Routing: Boosting the Efficiency of MoE-based LLMs
  7. 2025Faster MoE LLM Inference for Extremely Large Models
  8. 2025D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving