mlsanityData to 7 Oct 2026

FEPLB: Exploiting Copy Engines for Nearly Free MoE Load Balancing in Distributed Training

arXiv 2604.19654 ↗First posted 21 Apr 2026cs.DCcited by 1

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Fine-grained, per-micro-batch load balancing is essential for efficient Mixture-of-Experts (MoE) training, yet every prior dynamic scheduling scheme pays for it with extra communication that is hard to hide. Especially on modern bulk-transfer backends such as DeepEP. We make a simple but consequential observation: on the NVIDIA Hopper architecture the NVLink Copy Engine can move data between intra-node GPUs without consuming any SM cycles, effectively providing a nearly free communication channel that runs in parallel with compute kernels. FEPLB turns this idle hardware into a new parallel dimension for MoE load rebalancing. Its Two-Phase Dispatch first routes tokens across nodes via the standard EP backend, then redistributes dynamic-expert tokens and weights within the NVLink domain through the Copy Engine at nearly zero cost, while a lightweight CPU scheduler runs concurrently with static expert computation. Because FEPLB uses only Copy Engine and CPU that are orthogonal to those consumed by EP and PP, it coexists with existing parallel strategies without reconfiguration. On GLM-5's MoE layers (128 experts, no auxiliary loss, up to 16 H100 GPUs), FEPLB reduces the token straggler by 51-70% and the GEMM straggler by 50-68% with no measurable EP communication overhead. Its advantage grows with the EP degree: at EP=8, it achieves 2x lower token straggler than FasterMoE.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 8 of its 19 references that are in the map. First posted on arXiv 21 Apr 2026.

Builds on

8 of 19 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025FUSCO: High-Performance Distributed Data Shuffling via Transformation-Communication Fusion26 Dec
  2. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  3. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 5

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  4. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  5. 2025Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compilercited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026TAOT: Topology-Aware Optimal Transport for Dynamic Expert Replica Placement in MoE Trainingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026TopoEP: Topology-Aware Load Balancing for Expert-Parallel MoE Training
  2. 2026UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing
  3. 2026TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes
  4. 2025Fine-grained MoE Load Balancing with Linear Programming
  5. 2025MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing
  6. 2025RailS: Load Balancing for All-to-All Communication in Distributed Mixture-of-Experts Training
  7. 2025Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens
  8. 2026Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling