mlsanityData to 14 Oct 2026

Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placement

arXiv 2607.08782 ↗First posted 13 Jun 2026cs.LG cs.AIcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Expert parallelism has become the prevailing paradigm to serve Mixture-of-Experts (MoE) models. Its efficiency depends on the communication and computation latencies of the GPUs, which are linked to the placement of experts in the GPUs. Existing works for optimizing expert placement focus on leveraging past requests' expert activation patterns. However, they demonstrate deficiencies facing diverse and rapidly changing request patterns, calling for an online, proactive approach. Implementing such an approach requires addressing several challenges: the uncertainty associated with incoming requests' expert activation, the cost of expert migration, and the NP-hard complexity in optimization. Therefore, we present Director, a new distributed MoE serving system that minimizes end-to-end latency via prediction-driven, online expert placement. Director uses either a lightweight cascaded predictor or a low-bit quantized replica for expert activation patterns of incoming requests. An online migration module then enacts the changes with near-zero downtime by executing migrations in compute-bound phases, keeping disruption bounded. At its core, a relaxation-based expert placement optimizer operates under capacity constraints, runs in polynomial time, and achieves a (1+ε) approximation ratio. Finally, we implement a prototype and demonstrate, through extensive experiments, a reduction in end-to-end latency of 11~55% for popular MoE models (e.g., Mistral, DeepSeek and Qwen) compared to existing work.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 30 of its 46 references that are in the map. First posted on arXiv 13 Jun 2026.

Builds on

30 of 46 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving17 Apr
  2. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing10 Feb
  3. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline9 Feb
  4. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models18 Jan
  5. 2024Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation23 Nov
  6. 2024Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules30 Jun
  7. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models29 Oct
  8. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  9. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 21

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2023Let's Verify Step by Stepcited by 3,789
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2023Qwen Technical Reportcited by 3,344
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2024Qwen2 Technical Reportcited by 2,120
  9. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  10. 2024Mixtral of Expertscited by 1,823
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
9 more
  1. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  3. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  4. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  5. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 53
  6. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 53
  7. 2023TA-MoE: Topology-Aware Large Scale Mixture-of-Expert Trainingcited by 15
  8. 2024Training and Serving System of Foundation Models: A Comprehensive Surveycited by 11
  9. 2015Approximation-Friendly Discrepancy Roundingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design
  2. 2026Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling
  3. 2025Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens
  4. 2026Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference
  5. 2025Toward Cost-Efficient Serving of Mixture-of-Experts with Asynchrony
  6. 2026ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernels
  7. 2026Moebius: Serving Mixture-of-Expert Models with Seamless Runtime Parallelism Switch
  8. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication