mlsanityData to 7 Oct 2026

LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training

arXiv 2602.11686 ↗First posted 12 Feb 2026cs.DC cs.LGcited by 5

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Expert parallelism is vital for effectively training Mixture-of-Experts (MoE) models, enabling different devices to host distinct experts, with each device processing different input data. However, during expert parallel training, dynamic routing results in significant load imbalance among experts: a handful of overloaded experts hinder overall iteration, emerging as a training bottleneck. In this paper, we introduce LAER-MoE, an efficient MoE training framework. The core of LAER-MoE is a novel parallel paradigm, Fully Sharded Expert Parallel (FSEP), which fully partitions each expert parameter by the number of devices and restores partial experts at expert granularity through All-to-All communication during training. This allows for flexible re-layout of expert parameters during training to enhance load balancing. In particular, we perform fine-grained scheduling of communication operations to minimize communication overhead. Additionally, we develop a load balancing planner to formulate re-layout strategies of experts and routing schemes for tokens during training. We perform experiments on an A100 cluster, and the results indicate that our system achieves up to 1.69x acceleration compared to the current state-of-the-art training systems. Source code available at https://github.com/PKU-DAIR/Hetu-Galvatron/tree/laer-moe.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 28 of its 49 references that are in the map. First posted on arXiv 12 Feb 2026.

Builds on

28 of 49 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production16 May
  2. 2025MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core21 Apr
  3. 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts27 Feb
  4. 2024Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping30 Apr
  5. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  6. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  7. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  8. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services20 May
  9. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  7. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  8. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  9. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  10. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  11. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  12. 2021Hash Layers For Large Sparse Modelscited by 236
7 more
  1. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  2. 2020PyTorch Distributed: Experiences on Accelerating Data Parallel Trainingcited by 87
  3. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 52
  4. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  5. 2024The SCIP Optimization Suite 9.0cited by 37
  6. 2023Improving Automatic Parallel Training via Balanced Memory Workload Optimizationcited by 13
  7. 2023OpenBA: An Open-sourced 15B Bilingual Asymmetric seq2seq Model Pre-trained from Scratchcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancingcited by 8
  2. 2026ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learningcited by 2
  3. 2026TAOT: Topology-Aware Optimal Transport for Dynamic Expert Replica Placement in MoE Trainingcited by 1
  4. 2026Breaking the Training Barrier of Billion-Parameter Universal Machine Learning Interatomic Potentialscited by 0
  5. 2026Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatorycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design
  2. 2026Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling
  3. 2026Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts
  4. 2022SMILE: Scaling Mixture-of-Experts with Efficient Bi-level Routing
  5. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
  6. 2026Memory-Efficient Expert Routing for Distributed MoE Training
  7. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
  8. 2026UniPool: A Globally Shared Expert Pool for Mixture-of-Experts