mlsanityData to 14 Oct 2026

MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism

arXiv 2506.22175 ↗First posted 27 Jun 2025cs.DCcited by 1

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Recently, Mixture-of-Experts (MoE) has become one of the most popular techniques to scale pre-trained models to extraordinarily large sizes. Dynamic activation of experts allows for conditional computation, increasing the number of parameters of neural networks, which is critical for absorbing the vast amounts of knowledge available in many deep learning areas. However, despite the existing system and algorithm optimizations, there are significant challenges to be tackled when it comes to the inefficiencies of communication and memory consumption. In this paper, we present the design and implementation of MPipeMoE, a high-performance library that accelerates MoE training with adaptive and memory-efficient pipeline parallelism. Inspired by that the MoE training procedure can be divided into multiple independent sub-stages, we design adaptive pipeline parallelism with an online algorithm to configure the granularity of the pipelining. Further, we analyze the memory footprint breakdown of MoE training and identify that activations and temporary buffers are the primary contributors to the overall memory footprint. Toward memory efficiency, we propose memory reusing strategies to reduce memory requirements by eliminating memory redundancies, and develop an adaptive selection component to determine the optimal strategy that considers both hardware capacities and model characteristics at runtime. We implement MPipeMoE upon PyTorch and evaluate it with common MoE models in a physical cluster consisting of 8 NVIDIA DGX A100 servers. Compared with the state-of-art approach, MPipeMoE achieves up to 2.8x speedup and reduces memory footprint by up to 47% in training large models.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 26 of its 42 references that are in the map. First posted on arXiv 27 Jun 2025.

Builds on

26 of 42 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  2. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  11. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  12. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
12 more
  1. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
  2. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  3. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  4. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  5. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 117
  6. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  7. 2022Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUEcited by 24
  8. 2022Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformerscited by 22
  9. 2022E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generationcited by 19
  10. 2021AxoNN: An asynchronous, message-driven parallel framework for extreme-scale deep learningcited by 15
  11. 2019Buddy Compression: Enabling Larger Memory for Deep Learning and HPC Workloads on GPUscited by 11
  12. 2022BLISS: Robust Sequence-to-Sequence Learning via Self-Supervised Input Representationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Expertscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts
  2. 2023Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism
  3. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
  4. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
  5. 2026EMO: Frustratingly Easy Progressive Training of Extendable MoE
  6. 2021FastMoE: A Fast Mixture-of-Expert Training System
  7. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training
  8. 2026Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism