mlsanityData to 7 Oct 2026

Scalable Training of Mixture-of-Experts Models with Megatron Core

arXiv 2603.07685 ↗First posted 8 Mar 2026cs.DC cs.CL cs.LGcited by 16

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Scaling Mixture-of-Experts (MoE) training introduces systems challenges absent in dense models. Because each token activates only a subset of experts, this sparsity allows total parameters to grow much faster than per-token computation, creating coupled constraints across memory, communication, and computation. Optimizing one dimension often shifts pressure to another, demanding co-design across the full system stack. We address these challenges for MoE training through integrated optimizations spanning memory (fine-grained recomputation, offloading, etc.), communication (optimized dispatchers, overlapping, etc.), and computation (Grouped GEMM, fusions, CUDA Graphs, etc.). The framework also provides Parallel Folding for flexible multi-dimensional parallelism, low-precision training support for FP8 and NVFP4, and efficient long-context training. On NVIDIA GB300 and GB200, it achieves 1,233/1,048 TFLOPS/GPU for DeepSeek-V3-685B and 974/919 TFLOPS/GPU for Qwen3-235B. As a performant, scalable, and production-ready open-source solution, it has been used across academia and industry for training MoE models ranging from billions to trillions of parameters on clusters scaling up to thousands of GPUs. This report explains how these techniques work, their trade-offs, and their interactions at the systems level, providing practical guidance for scaling MoE models with Megatron Core.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 73 of its 104 references that are in the map. First posted on arXiv 8 Mar 2026.

Builds on

73 of 104 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts26 Jan
  2. 2025MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core21 Apr
  3. 2025Muon is Scalable for LLM Training24 Feb

Other references · 70

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2025Qwen3 Technical Reportcited by 8,195
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  9. 2024Qwen2.5 Technical Reportcited by 4,713
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
38 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2024Mixtral of Expertscited by 1,785
  6. 2020GLU Variants Improve Transformercited by 1,724
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  10. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  11. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  12. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  13. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  14. 2024DeepSeek-V3 Technical Reportcited by 724
  15. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  16. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  17. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  18. 2021A White Paper on Neural Network Quantizationcited by 453
  19. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  20. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  21. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  22. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  23. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  24. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  25. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  26. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  27. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  28. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  29. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 191
  30. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  31. 2022FP8 Formats for Deep Learningcited by 187
  32. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
  33. 2023Microscaling Data Formats for Deep Learningcited by 132
  34. 2022FP8 Quantization: The Power of the Exponentcited by 87
  35. 2023FP8-LM: Training FP8 Large Language Modelscited by 75
  36. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  37. 2025Pretraining Large Language Models with NVFP4cited by 67
  38. 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 67

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 16.

16 papers.

Sort:

By area:

  1. 2026Kimi K3: Open Frontier Intelligencecited by 99
  2. 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 32
  3. 2026UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancingcited by 8
  4. 2026HiFloat4 Format for Language Model Pre-training on Ascend NPUscited by 4
  5. 2026Expert-Choice Routing Enables Adaptive Computation in Diffusion Language Modelscited by 3
  6. 2026OrderMoE: An expert similarity driven distributed edge MoE inferencecited by 3
  7. 2026Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Trainingcited by 2
  8. 2026DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72cited by 1
  9. 2026Resilient AI Supercomputer Networking using MRC and SRv6cited by 0
  10. 2026EMO: Frustratingly Easy Progressive Training of Extendable MoEcited by 0
  11. 2026How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterizationcited by 0
  12. 2026AdaptiveLoad: Towards Efficient Video Diffusion Transformer Trainingcited by 0
4 more
  1. 2026Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configurationcited by 0
  2. 2026Mixture-of-Kittens: MoE Megakernel for NVL72scited by 0
  3. 2026MoRE: Scaling mixture of experts with hardware-aware low-rank routingcited by 0
  4. 2026Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
  2. 2025X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms
  3. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production
  4. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
  5. 2026MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models
  6. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training
  7. 2022Tutel: Adaptive Mixture-of-Experts at Scale
  8. 2024Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation