mlsanityData to 7 Oct 2026

FastMoE: A Fast Mixture-of-Expert Training System

arXiv 2103.13262 ↗First posted 24 Mar 2021cs.LG cs.CL cs.DCcited by 110

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Expert (MoE) presents a strong potential in enlarging the size of language model to trillions of parameters. However, training trillion-scale MoE requires algorithm and system co-design for a well-tuned high performance distributed training system. Unfortunately, the only existing platform that meets the requirements strongly depends on Google's hardware (TPU) and software (Mesh Tensorflow) stack, and is not open and available to the public, especially GPU and PyTorch communities. In this paper, we present FastMoE, a distributed MoE training system based on PyTorch with common accelerators. The system provides a hierarchical interface for both flexible model design and easy adaption to different applications, such as Transformer-XL and Megatron-LM. Different from direct implementation of MoE models using PyTorch, the training speed is highly optimized in FastMoE by sophisticated high-performance acceleration skills. The system supports placing different experts on multiple GPUs across multiple nodes, enabling enlarging the number of experts linearly against the number of GPUs. The source of FastMoE is available at https://github.com/laekov/fastmoe under Apache-2 license.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 16 of its 20 references that are in the map. First posted on arXiv 24 Mar 2021.

Builds on

16 of 20 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
4 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  3. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  4. 2018Tensor2Tensor for Neural Machine Translationcited by 85

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 14; 2023: 18; 2024: 22; 2025: 30; 2026: 21.

110 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023A Comprehensive Overview of Large Language Modelscited by 639
  3. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  4. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  5. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  6. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  7. 2023Efficient Large Language Models: A Surveycited by 183
  8. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  9. 2022CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecastingcited by 143
  10. 2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Pathscited by 141
  11. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 141
  12. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
38 more
  1. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 101
  2. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 58
  3. 2021CPM-2: Large-scale Cost-effective Pre-trained Language Modelscited by 52
  4. 2023OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllerscited by 47
  5. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  6. 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Expertscited by 46
  7. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Modelscited by 43
  8. 2022Residual Mixture of Expertscited by 39
  9. 2022Mask and Reason: Pre-Training Knowledge Graph Transformers for Complex Logical Queriescited by 39
  10. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Servicescited by 37
  11. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training Systemcited by 36
  12. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 36
  13. 2022Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Modelscited by 31
  14. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 31
  15. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizercited by 28
  16. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inferencecited by 28
  17. 2021SpeechMoE2: Mixture-of-Experts Model with Improved Routingcited by 26
  18. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routingcited by 26
  19. 2023Lifting the Curse of Capacity Gap in Distilling Language Modelscited by 25
  20. 2022A Survey on Dynamic Neural Networks for Natural Language Processingcited by 24
  21. 2023Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Expertscited by 23
  22. 2021M6-T: Exploring Sparse Expert Models and Beyondcited by 22
  23. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  24. 2023MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Expertscited by 21
  25. 2024LocMoE: A Low-Overhead MoE for Large Language Model Trainingcited by 18
  26. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Modelscited by 17
  27. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  28. 2023TA-MoE: Topology-Aware Large Scale Mixture-of-Expert Trainingcited by 15
  29. 2023ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformercited by 15
  30. 2023Experts Weights Averaging: A New General Training Scheme for Vision Transformerscited by 12
  31. 2025V2-SAM: Marrying SAM2 with Multi-Prompt Experts for Cross-View Object Correspondencecited by 12
  32. 2024Training and Serving System of Foundation Models: A Comprehensive Surveycited by 11
  33. 2025eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inferencecited by 11
  34. 2024Norface: Improving Facial Expression Analysis by Identity Normalizationcited by 10
  35. 2022Neural Implicit Dictionary via Mixture-of-Expert Trainingcited by 9
  36. 2025Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Schedulingcited by 9
  37. 2025Faster MoE LLM Inference for Extremely Large Modelscited by 9
  38. 2024Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Schedulingcited by 7

50 most cited of 110 All 110 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUs
  2. 2025HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap
  3. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations
  4. 2024Mixture of Diverse Size Experts
  5. 2025FlashMoE: Fast Distributed MoE in a Single Kernel
  6. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production
  7. 2023Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism
  8. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading