mlsanityData to 13 Oct 2026

UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing

arXiv 2606.04101 ↗First posted 2 Jun 2026cs.DC cs.LGcited by 8

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Large-scale expert parallelism (EP) is becoming pivotal for training and serving frontier MoE models, but it also amplifies device-level expert load imbalance into compute stragglers, token all-to-all bottlenecks, and activation-memory spikes. Existing balancers redistribute experts periodically based on historical load, which becomes unreliable for production deployments with non-stationary load patterns. We present UltraEP, the first exact-load, real-time balancer for large-EP MoE training and serving prefill on rack-scale nodes (RSNs). Leveraging the extended scale-up connectivity among dozens of GPUs within RSNs, UltraEP rebalances every microbatch and layer on critical paths, which requires nontrivial co-design of plan solving and expert replication communication to minimize exposed overhead. To this end, UltraEP eagerly reacts to post-gating load with an efficient quota-driven planner, and executes the resulting irregular expert-state transfers with RSN-native persistent tile streaming and relay-based fan-out mitigation. We evaluate UltraEP in a multi-RSN deployment of up to 256 GPUs, using cutting-edge MoE models from 106B to 671B parameters. Averaged across training and serving, UltraEP achieves 94.3% of the force-balanced ideal throughput, delivering 1.49× improvement over no-balancing, while reducing the final inter-rank imbalance from 1.30-4.01 to 1.01-1.04.

Key facts. Cited by 8 papers in the map (data through 13 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 35 of its 69 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

35 of 69 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026Scalable Training of Mixture-of-Experts Models with Megatron Core8 Mar
  2. 2026LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training12 Feb
  3. 2026PROBE: Co-Balancing Computation and Communication in MoE Inference via Real-Time Predictive Prefetching31 Jan
  4. 2025UCCL-EP: Portable Expert-Parallel Communication22 Dec
  5. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production16 May
  6. 2025MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core21 Apr
  7. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models18 Jan
  8. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  9. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  10. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 25

  1. 2025Qwen3 Technical Reportcited by 8,510
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,831
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,463
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,274
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,044
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,804
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,729
  8. 2024Qwen2 Technical Reportcited by 2,102
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,851
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,618
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,386
  12. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,319
13 more
  1. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,289
  2. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,117
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 896
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 853
  5. 2016Training Deep Nets with Sublinear Memory Costcited by 840
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 626
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 602
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 545
  9. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 514
  10. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 200
  11. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  12. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  13. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 53

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Kimi K3: Open Frontier Intelligencecited by 122
  2. 2026ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernelscited by 1
  3. 2026HCRMap: Pressure-Aware Hot-Expert Residency Mapping for 3.5D MoE Chiplet Inferencecited by 0
  4. 2026Searching for Plans You Can Actually Build: A Realizability-Aware Full-Space Optimizer for MoE Training and Servingcited by 0
  5. 2026RoutePack: Expert Placement and Attention-Aware Data Packing for MoE Reinforcement Learningcited by 0
  6. 2026TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimescited by 0
  7. 2026FreeBalance: Pre-Routing Online Moe Load Balancing via Residual Workload Predictioncited by 0
  8. 2026TopoEP: Topology-Aware Load Balancing for Expert-Parallel MoE Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FEPLB: Exploiting Copy Engines for Nearly Free MoE Load Balancing in Distributed Training
  2. 2026Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts
  3. 2025Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens
  4. 2026Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling
  5. 2026Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism
  6. 2026EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference
  7. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs
  8. 2026UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpods