mlsanityData to 7 Oct 2026

HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System

arXiv 2203.14685 ↗First posted 28 Mar 2022cs.DCcited by 36

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

As giant dense models advance quality but require large amounts of GPU budgets for training, the sparsely gated Mixture-of-Experts (MoE), a kind of conditional computation architecture, is proposed to scale models while keeping their computation constant. Specifically, the input tokens are routed by the gate network and only activates part of the expert network. Existing MoE training systems only support part of mainstream MoE models (e.g. Top k) training under expensive high-bandwidth GPU clusters. In this paper, we present HetuMoE, a high-performance large-scale sparse MoE training system built on Hetu. HetuMoE provides multiple gating strategies and efficient GPU kernel implementations. To further improve the training efficiency on commodity GPU clusters (e.g, with only 1 NiC), we introduce the hierarchical AllToAll communication that combines hierarchical networks and aggregating messages. Compared with existing state-of-the-art MoE systems, HetuMoE obtains at least 15% speedup. Specifically, HetuMoE outperforms DeepSpeed-MoE up to 8.1x under the switch gate with a batch size of 32. Our code is available at: https://github.com/PKU-DAIR/Hetu.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 17 of its 22 references that are in the map. First posted on arXiv 28 Mar 2022.

Builds on

17 of 22 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan
  2. 2021M6-T: Exploring Sparse Expert Models and Beyond31 May
  3. 2021FastMoE: A Fast Mixture-of-Expert Training System24 Mar

Other references · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  10. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  11. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  12. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
2 more
  1. 2021Hash Layers For Large Sparse Modelscited by 236
  2. 2021Towards More Effective and Economic Sparsely-Activated Modelcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 14; 2025: 8; 2026: 7.

36 papers.

Sort:

By area:

  1. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  2. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  3. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 56
  4. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 52
  5. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  6. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Trainingcited by 47
  7. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  8. 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Expertscited by 46
  9. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 31
  10. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inferencecited by 28
  11. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Productioncited by 28
  12. 2024Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlappingcited by 22
24 more
  1. 2024LocMoE: A Low-Overhead MoE for Large Language Model Trainingcited by 18
  2. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Modelscited by 17
  3. 2023TA-MoE: Topology-Aware Large Scale Mixture-of-Expert Trainingcited by 15
  4. 2023Improving Automatic Parallel Training via Balanced Memory Workload Optimizationcited by 13
  5. 2023Experts Weights Averaging: A New General Training Scheme for Vision Transformerscited by 12
  6. 2025eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inferencecited by 11
  7. 2024Lazarus: Resilient and Elastic Training of Mixture-of-Experts Modelscited by 9
  8. 2024Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedulescited by 7
  9. 2024Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Schedulingcited by 7
  10. 2025HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUscited by 7
  11. 2024EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inferencecited by 6
  12. 2025Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Designcited by 6
  13. 2024LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashingcited by 5
  14. 2025SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decouplingcited by 5
  15. 2024Pro-Prophet: A Systematic Load Balancing Method for Efficient Parallel Training of Large-scale MoE Modelscited by 4
  16. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applicationscited by 2
  17. 2026ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learningcited by 2
  18. 2025Towards Efficient Federated Learning of Networked Mixture-of-Experts for Mobile Edge Computingcited by 1
  19. 2024MoNTA: Accelerating Mixture-of-Experts Training with Network-Traffc-Aware Parallel Optimizationcited by 0
  20. 2026Mixture-of-Experts in Remote Sensing: A Surveycited by 0
  21. 2026Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelismcited by 0
  22. 2026Accelerating MoE with Dynamic In-Switch Computing on Multi-GPUscited by 0
  23. 2026ID Balancing: Stable Training of Extremely Sparse MoE via PID-Based Load Controlcited by 0
  24. 2026Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation
  2. 2026Scalable Training of Mixture-of-Experts Models with Megatron Core
  3. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
  4. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
  5. 2026Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts
  6. 2026MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models
  7. 2025X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms
  8. 2026Mixture-of-Kittens: MoE Megakernel for NVL72s