mlsanityData to 7 Oct 2026

Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules

arXiv 2407.00599 ↗First posted 30 Jun 2024cs.DC cs.LGcited by 7

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Sparsely-activated Mixture-of-Expert (MoE) layers have found practical applications in enlarging the model size of large-scale foundation models, with only a sub-linear increase in computation demands. Despite the wide adoption of hybrid parallel paradigms like model parallelism, expert parallelism, and expert-sharding parallelism (i.e., MP+EP+ESP) to support MoE model training on GPU clusters, the training efficiency is hindered by communication costs introduced by these parallel paradigms. To address this limitation, we propose Parm, a system that accelerates MP+EP+ESP training by designing two dedicated schedules for placing communication tasks. The proposed schedules eliminate redundant computations and communications and enable overlaps between intra-node and inter-node communications, ultimately reducing the overall training time. As the two schedules are not mutually exclusive, we provide comprehensive theoretical analyses and derive an automatic and accurate solution to determine which schedule should be applied in different scenarios. Experimental results on an 8-GPU server and a 32-GPU cluster demonstrate that Parm outperforms the state-of-the-art MoE training system, DeepSpeed-MoE, achieving 1.13× to 5.77× speedup on 1296 manually configured MoE layers and approximately 3× improvement on two real-world MoE models based on BERT and GPT-2.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 22 of its 36 references that are in the map. First posted on arXiv 30 Jun 2024.

Builds on

22 of 36 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training11 Mar
  2. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  3. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  4. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System28 Mar
  5. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  10. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  11. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  12. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
5 more
  1. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  2. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  3. 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 55
  4. 2018SparCML: High-Performance Sparse Communication for Machine Learningcited by 34
  5. 2022Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformerscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 1.

7 papers.

  1. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  2. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Modelscited by 17
  3. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 17
  4. 2025SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decouplingcited by 5
  5. 2025HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swapcited by 4
  6. 2025PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMscited by 0
  7. 2026Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation
  2. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
  3. 2026Scalable Training of Mixture-of-Experts Models with Megatron Core
  4. 2021M6-T: Exploring Sparse Expert Models and Beyond
  5. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production
  6. 2025FlashMoE: Fast Distributed MoE in a Single Kernel
  7. 2026MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models
  8. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts