# MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production

arXiv: https://arxiv.org/abs/2505.11432 · HTML: https://mlsanity.com/paper/2505.11432/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 35 of its 56 references that are in the map.
- First posted on arXiv 16 May 2025.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [Ring Attention with Blockwise Transformers for Near-Infinite Context](https://mlsanity.com/paper/2310.01889/) (2023)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022)
- [ZeRO-Offload: Democratizing Billion-Scale Model Training](https://mlsanity.com/paper/2101.06840/) (2021)
- [Reducing Activation Recomputation in Large Transformer Models](https://mlsanity.com/paper/2205.05198/) (2022)
- [ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning](https://mlsanity.com/paper/2104.07857/) (2021)
- [DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models](https://mlsanity.com/paper/2309.14509/) (2023)
- [Tutel: Adaptive Mixture-of-Experts at Scale](https://mlsanity.com/paper/2206.03382/) (2022)
- [Sequence Parallelism: Long Sequence Training from System Perspective](https://mlsanity.com/paper/2105.13120/) (2021)
- [MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs](https://mlsanity.com/paper/2402.15627/) (2024)
- [Accelerating Distributed MoE Training and Inference with Lina](https://mlsanity.com/paper/2210.17223/) (2022)
- [TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training](https://mlsanity.com/paper/2410.06511/) (2024)
- [FP8-LM: Training FP8 Large Language Models](https://mlsanity.com/paper/2310.18313/) (2023)
- [FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion](https://mlsanity.com/paper/2406.06858/) (2024)
- [Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts](https://mlsanity.com/paper/2502.19811/) (2025)
- [Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads](https://mlsanity.com/paper/2105.05720/) (2021)

## Built on by (most cited first, 30 of 31)

- [UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing](https://mlsanity.com/paper/2606.04101/) (2026)
- [LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training](https://mlsanity.com/paper/2602.11686/) (2026)
- [UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Training](https://mlsanity.com/paper/2604.19241/) (2026)
- [Collective Communication for 100k+ GPUs](https://mlsanity.com/paper/2510.20171/) (2025)
- [Reliable and Resilient Collective Communication Library for LLM Training and Serving](https://mlsanity.com/paper/2512.25059/) (2025)
- [Towards Intrinsic Interpretability of Large Language Models:A Survey of Design Principles and Architectures](https://mlsanity.com/paper/2604.16042/) (2026)
- [A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System Challenges](https://mlsanity.com/paper/2507.21117/) (2025)
- [ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learning](https://mlsanity.com/paper/2605.08639/) (2026)
- [FlexLink: Boosting your NVLink Bandwidth by 27% without accuracy concern](https://mlsanity.com/paper/2510.15882/) (2025)
- [BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models](https://mlsanity.com/paper/2512.12131/) (2025)
- [CRAFT: Fine-Grained Cost-Aware Expert Replication For Efficient Mixture-of-Experts Serving](https://mlsanity.com/paper/2603.28768/) (2026)
- [Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns](https://mlsanity.com/paper/2604.23150/) (2026)
- [Distributed Training under Packet Loss](https://mlsanity.com/paper/2507.07114/) (2025)
- [MemFine: Memory-Aware Fine-Grained Scheduling for MoE Training](https://mlsanity.com/paper/2511.21431/) (2025)
- [ReaLB: Real-Time Load Balancing for Multimodal MoE Inference](https://mlsanity.com/paper/2604.19503/) (2026)
- [Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts](https://mlsanity.com/paper/2604.19835/) (2026)
- [Searching for Plans You Can Actually Build: A Realizability-Aware Full-Space Optimizer for MoE Training and Serving](https://mlsanity.com/paper/2607.18631/) (2026)
- [MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models](https://mlsanity.com/paper/2601.06857/) (2026)
- [Efficient Training on Multiple Consumer GPUs with RoundPipe](https://mlsanity.com/paper/2604.27085/) (2026)
- [MoE-Hub: Taming Software Complexity for Seamless MoE Overlap with Hardware-Accelerated Communication on Multi-GPU Systems](https://mlsanity.com/paper/2605.05888/) (2026)
- [SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training](https://mlsanity.com/paper/2608.11034/) (2026)
- [TerraceMoE: A Cost Model for Hierarchical MoE All-to-All Communication](https://mlsanity.com/paper/2608.27874/) (2026)
- [Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration](https://mlsanity.com/paper/2608.28511/) (2026)
- [Epoch: Compiling Diffusion Blocks for Sparse MoE Serving](https://mlsanity.com/paper/2609.09748/) (2026)
- [From Semantic to Token Communication: The Next Paradigm for Large-Model-Driven 6G Intelligent Connectivity](https://mlsanity.com/paper/2609.10714/) (2026)
- [mKernel: Fast Multi-GPU, Multi-Node Fused Kernels](https://mlsanity.com/paper/2609.13585/) (2026)
- [HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training](https://mlsanity.com/paper/2609.39350/) (2026)
- [Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUs](https://mlsanity.com/paper/2609.40093/) (2026)
- [CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training](https://mlsanity.com/paper/2610.05744/) (2026)
- [Memory-Efficient Expert Routing for Distributed MoE Training](https://mlsanity.com/paper/2610.07333/) (2026)

## Abstract (arXiv)

We present MegaScale-MoE, a production system tailored for the efficient training of large-scale mixture-of-experts (MoE) models. MoE emerges as a promising architecture to scale large language models (LLMs) to unprecedented sizes, thereby enhancing model performance. However, existing MoE training systems experience a degradation in training efficiency, exacerbated by the escalating scale of MoE models and the continuous evolution of hardware. Recognizing the pivotal role of efficient communication in enhancing MoE training, MegaScale-MoE customizes communication-efficient parallelism strategies for attention and FFNs in each MoE layer and adopts a holistic approach to overlap communication with computation at both inter- and intra-operator levels. Additionally, MegaScale-MoE applies communication compression with adjusted communication patterns to lower precision, further improving training efficiency. When training a 352B MoE model on 1,440 NVIDIA Hopper GPUs, MegaScale-MoE achieves a training throughput of 1.41M tokens/s, improving the efficiency by 1.88× compared to Megatron-LM. We share our operational experience in accelerating MoE training and hope that by offering our insights in system design, this work will motivate future research in MoE systems.