mlsanityData to 7 Oct 2026

FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models

arXiv 2501.10714 ↗First posted 18 Jan 2025cs.LGcited by 17

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Recent large language models (LLMs) have tended to leverage sparsity to reduce computations, employing the sparsely activated mixture-of-experts (MoE) technique. MoE introduces four modules, including token routing, token communication, expert computation, and expert parallelism, that impact model quality and training efficiency. To enable versatile usage of MoE models, we introduce FSMoE, a flexible training system optimizing task scheduling with three novel techniques: 1) Unified abstraction and online profiling of MoE modules for task scheduling across various MoE implementations. 2) Co-scheduling intra-node and inter-node communications with computations to minimize communication overheads. 3) To support near-optimal task scheduling, we design an adaptive gradient partitioning method for gradient aggregation and a schedule to adaptively pipeline communications and computations. We conduct extensive experiments with configured MoE layers and real-world MoE models on two GPU clusters. Experimental results show that 1) our FSMoE supports four popular types of MoE routing functions and is more efficient than existing implementations (with up to a 1.42× speedup), and 2) FSMoE outperforms the state-of-the-art MoE training systems (DeepSpeed-MoE and Tutel) by 1.18×-1.22× on 1458 MoE layers and 1.19×-3.01× on real-world MoE models based on GPT-2 and Mixtral using a popular routing function.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 33 of its 55 references that are in the map. First posted on arXiv 18 Jan 2025.

Builds on

33 of 55 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules30 Jun
  2. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training11 Mar
  3. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  4. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  5. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System28 Mar
  6. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan
  7. 2021FastMoE: A Fast Mixture-of-Expert Training System24 Mar

Other references · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  7. 2024Mixtral of Expertscited by 1,785
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  10. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  11. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  12. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
14 more
  1. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  2. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  3. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  4. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  5. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 98
  6. 2018Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutescited by 63
  7. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 53
  8. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 52
  9. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Expertscited by 48
  10. 2021Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloadscited by 40
  11. 2023PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformationcited by 25
  12. 2022Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformerscited by 22
  13. 2024T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectivescited by 15
  14. 2023Experts Weights Averaging: A New General Training Scheme for Vision Transformerscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 9.

17 papers.

Sort:

By area:

  1. 2026UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancingcited by 8
  2. 2025SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decouplingcited by 5
  3. 2025DynaMo: Runtime Switchable Quantization for MoE with Cross-Dataset Adaptationcited by 4
  4. 2025HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swapcited by 4
  5. 2025FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Trainingcited by 2
  6. 2025HybridEP: Scaling Expert Parallelism to Cross-Datacenter Scenario via Hybrid Expert/Data Transmissioncited by 2
  7. 202510Cache: Heterogeneous Resource-Aware Tensor Caching and Migration for LLM Trainingcited by 2
  8. 2025Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelismcited by 1
  9. 2026ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Trainingcited by 1
  10. 2026Multi-level context Modeling for consistent expert selection in Mixture-of-Expertscited by 1
  11. 2025PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMscited by 0
  12. 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Schedulingcited by 0
5 more
  1. 2026CogniRoute: Learning to Route Social Evidence in Omni-Modal Modelscited by 0
  2. 2026Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placementcited by 0
  3. 2026X-Stage: Modeling Post-Issue Backpressure in GPU Communication--Computation Fusioncited by 0
  4. 2026The Life of a Token: from Words to Bits on the Wirecited by 0
  5. 2026Mixture-of-Kittens: MoE Megakernel for NVL72scited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  2. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  3. 2024Mixture of Diverse Size Experts
  4. 2024Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation
  5. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline
  6. 2025DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs
  7. 2026Mixture of Heterogeneous Grouped Experts for Language Modeling
  8. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource