mlsanityData to 14 Oct 2026

MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production

arXiv 2505.11432 ↗First posted 16 May 2025cs.LG cs.DCcited by 31

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

We present MegaScale-MoE, a production system tailored for the efficient training of large-scale mixture-of-experts (MoE) models. MoE emerges as a promising architecture to scale large language models (LLMs) to unprecedented sizes, thereby enhancing model performance. However, existing MoE training systems experience a degradation in training efficiency, exacerbated by the escalating scale of MoE models and the continuous evolution of hardware. Recognizing the pivotal role of efficient communication in enhancing MoE training, MegaScale-MoE customizes communication-efficient parallelism strategies for attention and FFNs in each MoE layer and adopts a holistic approach to overlap communication with computation at both inter- and intra-operator levels. Additionally, MegaScale-MoE applies communication compression with adjusted communication patterns to lower precision, further improving training efficiency. When training a 352B MoE model on 1,440 NVIDIA Hopper GPUs, MegaScale-MoE achieves a training throughput of 1.41M tokens/s, improving the efficiency by 1.88× compared to Megatron-LM. We share our operational experience in accelerating MoE training and hope that by offering our insights in system design, this work will motivate future research in MoE systems.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 35 of its 56 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

35 of 56 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts27 Feb
  2. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  3. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  4. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System28 Mar
  5. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2024Mixtral of Expertscited by 1,823
  9. 2020GLU Variants Improve Transformercited by 1,804
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  12. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
18 more
  1. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  2. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  3. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  4. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  5. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  6. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  7. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  8. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  9. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 126
  10. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  11. 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 85
  12. 2023FP8-LM: Training FP8 Large Language Modelscited by 77
  13. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 57
  14. 2021Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloadscited by 40
  15. 2023DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Trainingcited by 29
  16. 2018TicTac: Accelerating Distributed Deep Learning with Communication Schedulingcited by 27
  17. 2025TileLink: Generating Efficient Compute-Communication Overlapping Kernels using Tile-Centric Primitivescited by 24
  18. 2024T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectivescited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 24.

31 papers.

Sort:

By area:

  1. 2026UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancingcited by 9
  2. 2026LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Trainingcited by 6
  3. 2026UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Trainingcited by 5
  4. 2025Collective Communication for 100k+ GPUscited by 4
  5. 2025Reliable and Resilient Collective Communication Library for LLM Training and Servingcited by 4
  6. 2026Towards Intrinsic Interpretability of Large Language Models:A Survey of Design Principles and Architecturescited by 4
  7. 2025A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System Challengescited by 3
  8. 2026ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learningcited by 3
  9. 2025FlexLink: Boosting your NVLink Bandwidth by 27% without accuracy concerncited by 2
  10. 2025BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Modelscited by 2
  11. 2026CRAFT: Fine-Grained Cost-Aware Expert Replication For Efficient Mixture-of-Experts Servingcited by 2
  12. 2026Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patternscited by 2
19 more
  1. 2025Distributed Training under Packet Losscited by 1
  2. 2025MemFine: Memory-Aware Fine-Grained Scheduling for MoE Trainingcited by 1
  3. 2026ReaLB: Real-Time Load Balancing for Multimodal MoE Inferencecited by 1
  4. 2026Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Expertscited by 1
  5. 2026Searching for Plans You Can Actually Build: A Realizability-Aware Full-Space Optimizer for MoE Training and Servingcited by 1
  6. 2026MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Modelscited by 0
  7. 2026Efficient Training on Multiple Consumer GPUs with RoundPipecited by 0
  8. 2026MoE-Hub: Taming Software Complexity for Seamless MoE Overlap with Hardware-Accelerated Communication on Multi-GPU Systemscited by 0
  9. 2026SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Trainingcited by 0
  10. 2026TerraceMoE: A Cost Model for Hierarchical MoE All-to-All Communicationcited by 0
  11. 2026Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configurationcited by 0
  12. 2026Epoch: Compiling Diffusion Blocks for Sparse MoE Servingcited by 0
  13. 2026From Semantic to Token Communication: The Next Paradigm for Large-Model-Driven 6G Intelligent Connectivitycited by 0
  14. 2026mKernel: Fast Multi-GPU, Multi-Node Fused Kernelscited by 0
  15. 2026HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Trainingcited by 0
  16. 2026Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUscited by 0
  17. 2026CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Trainingcited by 0
  18. 2026Memory-Efficient Expert Routing for Distributed MoE Trainingcited by 0
  19. 2026Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism
  2. 2026Scalable Training of Mixture-of-Experts Models with Megatron Core
  3. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
  4. 2026Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts
  5. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
  6. 2025X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms
  7. 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models
  8. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations