X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms
On the map: Mixture-of-experts → Efficient MoE training and inference systems
Abstract (arXiv)
Emerging expert-specialized Mixture-of-Experts (MoE) architectures, such as DeepSeek-MoE, deliver strong model quality through fine-grained expert segmentation and large top-k routing. However, their scalability is limited by substantial activation memory overhead and costly all-to-all communication. Furthermore, current MoE training systems - primarily optimized for NVIDIA GPUs - perform suboptimally on non-NVIDIA platforms, leaving significant computational potential untapped. In this work, we present X-MoE, a novel MoE training system designed to deliver scalable training performance for next-generation MoE architectures. X-MoE achieves this via several novel techniques, including efficient padding-free MoE training with cross-platform kernels, redundancy-bypassing dispatch, and hybrid parallelism with sequence-sharded MoE blocks. Our evaluation on the Frontier supercomputer, powered by AMD MI250X GPUs, shows that X-MoE scales DeepSeek-style MoEs up to 545 billion parameters across 1024 GPUs - 10x larger than the largest trainable model with existing methods under the same hardware budget, while maintaining high training throughput. The source code of X-MoE is available at https://github.com/Supercomputing-System-AI-Lab/X-MoE.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 15 of its 33 references that are in the map. First posted on arXiv 18 Aug 2025.
Builds on
15 of 33 references.
Closest ancestors · 5
Same direction, newest first
- 2024Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping30 Apr
- 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training11 Mar
- 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
- 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
- 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan
Other references · 10
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
- 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
- 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
- 2016Training Deep Nets with Sublinear Memory Costcited by 825
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Scalable Training of Mixture-of-Experts Models with Megatron Core
- 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
- 2026Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism
- 2026Mixture-of-Kittens: MoE Megakernel for NVL72s
- 2026MoEBlaze: Breaking the Memory Wall for Efficient MoE Training on Modern GPUs
- 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production
- 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models
- 2024Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation