mlsanityData to 14 Oct 2026

Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism

arXiv 2304.11414 ↗First posted 22 Apr 2023cs.DC cs.LGcited by 4

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

The Mixture of Experts (MoE) model becomes an important choice of large language models nowadays because of its scalability with sublinear computational complexity for training and inference. However, existing MoE models suffer from two critical drawbacks, 1) tremendous inner-node and inter-node communication overhead introduced by all-to-all dispatching and gathering, and 2) limited scalability for the backbone because of the bound data parallel and expert parallel to scale in the expert dimension. In this paper, we systematically analyze these drawbacks in terms of training efficiency in the parallel framework view and propose a novel MoE architecture called Pipeline MoE (PPMoE) to tackle them. PPMoE builds expert parallel incorporating with tensor parallel and replaces communication-intensive all-to-all dispatching and gathering with a simple tensor index slicing and inner-node all-reduce. Besides, it is convenient for PPMoE to integrate pipeline parallel to further scale the backbone due to its flexible parallel architecture. Extensive experiments show that PPMoE not only achieves a more than 1.75× speed up compared to existing MoE architectures but also reaches 90% throughput of its corresponding backbone model that is 20× smaller.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 26 of its 36 references that are in the map. First posted on arXiv 22 Apr 2023.

Builds on

26 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
13 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  2. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  4. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  5. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  6. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  7. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  8. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  9. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  10. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 117
  11. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  12. 2022Formal Algorithms for Transformerscited by 72
  13. 2021Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model Trainingcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 0; 2025: 1; 2026: 1.

4 papers.

  1. 2023LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Modelscited by 72
  2. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 26
  3. 2025SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decouplingcited by 5
  4. 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Schedulingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration
  2. 2024EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference
  3. 2025FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training
  4. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline
  5. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  6. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference
  7. 2025Mixture of Experts in Large Language Models
  8. 2025Accelerating MoE Model Inference with Expert Sharding