mlsanityData to 7 Oct 2026

Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns

arXiv 2604.23150 ↗First posted 25 Apr 2026cs.LG cs.AI cs.ARcited by 2

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Most recent state-of-the-art (SOTA) large language models (LLMs) use Mixture-of-Experts (MoE) architectures to scale model capacity without proportional per-token compute, enabling higher-quality outputs at manageable serving costs. However, MoE inference at scale is fundamentally bottlenecked by expert load imbalance and inefficient token routing, especially in multi-node deployments where tokens are not guaranteed to be routed to local experts, resulting in significant inter-node all-to-all communication overhead. To systematically characterize these challenges, we profile SOTA open-source MoE models, including Llama 4 Maverick, DeepSeek V3-671B, and Qwen3-230B-A22B, on various datasets and collected over 100k real expert activation traces. Upon studying the expert activation patterns, we uncover various persistent properties across all the frontier MoE models: variable expert load imbalance, domain-specific expert activation where expert popularity shifts across task families (code, math, chat, general), and a strong correlation between prefill and decode expert activations. Motivated by these findings, we propose workload-aware micro-batch grouping and an expert placement strategy to maximize token locality to the destination expert, thereby reducing inter-node communication. Across models and datasets, these optimizations help reduce all2all communication data up to 20, resulting in lower MoE decode latency and better accelerator utilization.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 18 of its 27 references that are in the map. First posted on arXiv 25 Apr 2026.

Builds on

18 of 27 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models22 Feb
  2. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun

Other references · 15

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  8. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  9. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  10. 2023Fast Inference of Mixture-of-Experts Language Models with Offloadingcited by 88
  11. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cachecited by 48
  12. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Servicescited by 37
3 more
  1. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inferencecited by 28
  2. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Productioncited by 28
  3. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routingcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Servingcited by 2
  2. 2026Beyond Task-Agnostic: Task-Aware Grouping for Communication-Efficient Multi-Task MoE Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models
  2. 2025Advancing Expert Specialization for Better MoE
  3. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  4. 2026Slicing and Dicing: Configuring Optimal Mixtures of Experts
  5. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models
  6. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  7. 2025Mixture of Experts in Large Language Models
  8. 2026DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts