mlsanityData to 7 Oct 2026

MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing

arXiv 2506.07366 ↗First posted 9 Jun 2025cs.LG cs.ARcited by 4

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

In multi-GPU Mixture-of-Experts (MoE) network, experts are distributed across different GPUs, which creates load imbalance as each expert processes different number of tokens. Recent works improve MoE inference load balance by dynamically duplicating popular experts to more GPUs to process excessive tokens, which requires predicting the distribution before routing. In this paper, we discuss the tradeoff of prediction strategies, accuracies, overhead, and end-to-end system performance. We propose MoE-GPS, a framework that guides the selection of the optimal predictor design under various system configurations, by quantifying the performance impact to system-level model runtime. Specifically, we advocate for Distribution-Only Prediction, a prediction strategy that only predicts overall token distribution which significantly reduces overhead compared to the traditional Token-to-Expert Prediction. On Mixtral 8x7B MMLU dataset, MoE-GPS suggests Distribution-Only Prediction which improves end-to-end inference performance by more than 23% compared with Token-to-Expert Prediction.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 22 of its 36 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

22 of 36 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing25 Apr
  2. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models29 Oct
  3. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan
  4. 2021FastMoE: A Fast Mixture-of-Expert Training System24 Mar

Other references · 18

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2024Mixtral of Expertscited by 1,785
  10. 2020GLU Variants Improve Transformercited by 1,724
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
6 more
  1. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  2. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 191
  3. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  4. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 52
  5. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 36
  6. 2025TransMLA: Multi-Head Latent Attention Is All You Needcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Remoe: Towards Efficient and Low-Cost MoE Inference in Serverless Computingcited by 2
  2. 2026CRAFT: Fine-Grained Cost-Aware Expert Replication For Efficient Mixture-of-Experts Servingcited by 2
  3. 2026MoEless: Efficient MoE LLM Serving via Serverless Computingcited by 1
  4. 2025Mixture of Raytraced Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens
  2. 2026GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems
  3. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing
  4. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
  5. 2026TopoEP: Topology-Aware Load Balancing for Expert-Parallel MoE Training
  6. 2025HarMoEny: Efficient Multi-GPU Inference of MoE Models
  7. 2026ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving
  8. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging