mlsanityData to 7 Oct 2026

TopoEP: Topology-Aware Load Balancing for Expert-Parallel MoE Training

arXiv 2609.35481 ↗First posted 28 Sep 2026cs.DC cs.LGcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Dynamic routing creates severe load imbalance in large-scale expert-parallel Mixture-of-Experts (MoE) training, turning GPUs that host hot experts into stragglers. As each MoE layer waits for its slowest rank, these stragglers prolong the expert-parallel stage and reduce overall training efficiency. Existing expert-parallelism load-balancing (EPLB) systems commonly compute load-balancing plans on the CPU, incurring device--host data transfers and cross-rank synchronization that make scheduling at every layer and microbatch expensive. Their planning formulations also overlook the hierarchical communication costs of modern scale-up and scale-out GPU clusters. We present textitTopoEP, a GPU-native, topology-aware load-balancing system for large-scale MoE training. At each MoE layer and training microbatch, textitTopoEP converts the current routing result into hot-expert replication and token-rerouting decisions and executes the resulting plan without data-dependent host synchronization, reducing critical-path overhead. To generate these decisions, textitTopoEP uses a deterministic GPU solver that performs inter-node placement followed by intra-node refinement, allowing all ranks to independently produce bitwise-identical plans. On a 32-GPU NVIDIA H800 cluster, integrating textitTopoEP with Megatron-LM improves end-to-end training throughput by 6.2%--11.4% across three representative MoE models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 26 of its 35 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

26 of 35 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing2 Jun
  2. 2026Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts23 Jan
  3. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations16 Dec
  4. 2025SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decoupling28 Apr
  5. 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts27 Feb
  6. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  7. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  8. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  9. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  8. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  9. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  10. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  11. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  12. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 191
5 more
  1. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 134
  2. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 52
  3. 2021EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gatecited by 30
  4. 2022Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformerscited by 22
  5. 2025GPU-Initiated Networking for NCCLcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens
  2. 2026FEPLB: Exploiting Copy Engines for Nearly Free MoE Load Balancing in Distributed Training
  3. 2026Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling
  4. 2025MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing
  5. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing
  6. 2026TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes
  7. 2026ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving
  8. 2025HarMoEny: Efficient Multi-GPU Inference of MoE Models