mlsanityData to 14 Oct 2026

MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core

arXiv 2504.14960 ↗First posted 21 Apr 2025cs.LG cs.DCcited by 17

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture of Experts (MoE) models enhance neural network scalability by dynamically selecting relevant experts per input token, enabling larger model sizes while maintaining manageable computation costs. However, efficient training of large-scale MoE models across thousands of GPUs presents significant challenges due to limitations in existing parallelism strategies. We introduce an end-to-end training framework for large-scale MoE models that utilizes five-dimensional hybrid parallelism: Tensor Parallelism, Expert Parallelism, Context Parallelism, Data Parallelism, and Pipeline Parallelism. Central to our approach is MoE Parallel Folding, a novel strategy that decouples the parallelization of attention and MoE layers in Transformer models, allowing each layer type to adopt optimal parallel configurations. Additionally, we develop a flexible token-level dispatcher that supports both token-dropping and token-dropless MoE training across all five dimensions of parallelism. This dispatcher accommodates dynamic tensor shapes and coordinates different parallelism schemes for Attention and MoE layers, facilitating complex parallelism implementations. Our experiments demonstrate significant improvements in training efficiency and scalability. We achieve up to 49.3% Model Flops Utilization (MFU) for the Mixtral 8x22B model and 39.0% MFU for the Qwen2-57B-A14B model on H100 GPUs, outperforming existing methods. The framework scales efficiently up to 1,024 GPUs and maintains high performance with sequence lengths up to 128K tokens, validating its effectiveness for large-scale MoE model training. The code is available in Megatron-Core.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 24 of its 33 references that are in the map. First posted on arXiv 21 Apr 2025.

Builds on

24 of 33 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training11 Mar
  2. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  3. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  4. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services20 May
  5. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  8. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  9. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  10. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  11. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  12. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
7 more
  1. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  2. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  3. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  4. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
  5. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
  6. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 101
  7. 2024Upcycling Large Language Models into Mixture of Expertscited by 49

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 13.

17 papers.

Sort:

By area:

  1. 2026Scalable Training of Mixture-of-Experts Models with Megatron Corecited by 19
  2. 2026UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancingcited by 9
  3. 2026LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Trainingcited by 6
  4. 2026ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learningcited by 3
  5. 2026LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budgetcited by 3
  6. 2025WeChat-YATT: A Scalable, Simple, Efficient, and Production Ready Training Librarycited by 2
  7. 2025BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Modelscited by 2
  8. 2025G-Core: A Simple, Scalable and Balanced RLHF Trainercited by 1
  9. 2025DualSparse-MoE: Coordinating Tensor/Neuron-Level Sparsity with Expert Partition and Reconstructioncited by 1
  10. 2026MixServe: An Automatic Distributed Serving System for MoE Models with Hybrid Parallelism Based on Fused Communication Algorithmcited by 1
  11. 2026MoEless: Efficient MoE LLM Serving via Serverless Computingcited by 1
  12. 2026MinT: Managed Infrastructure for Training and Serving Millions of LLMscited by 1
5 more
  1. 2026Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inferencecited by 0
  2. 2026MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Modelscited by 0
  3. 2026Memory-Efficient Expert Routing for Distributed MoE Trainingcited by 0
  4. 2026Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shufflingcited by 0
  5. 2026ATTUNE: Agentic Throughput Tuning for LLM Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
  2. 2025Accelerating MoE Model Inference with Expert Sharding
  3. 2026Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism
  4. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing
  5. 2025X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms
  6. 2024Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation
  7. 2024LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing
  8. 2023Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism