mlsanityData to 7 Oct 2026

Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism

arXiv 2605.05049 ↗First posted 6 May 2026cs.DC cs.AI cs.LGcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Frontier models increasingly adopt Mixture-of-Experts (MoE) architectures to achieve large-model performance at reduced cost. However, training MoE models on HPC platforms is hindered by large memory footprints, frequent large-scale communication across heterogeneous networks, and severe workload imbalance. To characterize these challenges, we develop a mathematical model that quantifies memory, compute, and communication requirements for MoE configurations under various parallelization schemes, verified through micro-benchmarking, code instrumentation, and hardware profiling. Our analysis identifies performance bottlenecks: all-to-all latency at scale from expert parallelism, insufficient compute-communication overlap, low GPU utilization from imbalanced skinny GEMMs, and the absence of platform-aware hybrid parallelization strategies. To address these, we introduce Piper, a framework that leverages resource modeling to identify efficient training strategies for MoE models on target HPC platforms, applying pipeline parallelism with optimized schedules. Piper achieves 2-3.5X higher MFU than state-of-the-art frameworks such as X-MoE, and a novel all-to-all algorithm delivers 1.2-9X bandwidth over vendor implementation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 21 of its 32 references that are in the map. First posted on arXiv 6 May 2026.

Builds on

21 of 32 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  2. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System28 Mar
  3. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2024Mixtral of Expertscited by 1,785
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  11. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  12. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
6 more
  1. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  2. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  3. 2025Kimi K2: Open Agentic Intelligencecited by 523
  4. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  5. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 133
  6. 2023Zero Bubble Pipeline Parallelismcited by 39

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms
  2. 2025MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core
  3. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production
  4. 2026HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training
  5. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
  6. 2023Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism
  7. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
  8. 2026Scalable Training of Mixture-of-Experts Models with Megatron Core