mlsanityData to 7 Oct 2026

FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement

arXiv 2304.03946 ↗First posted 8 Apr 2023cs.DC cs.LGcited by 52

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

With the increasing data volume, there is a trend of using large-scale pre-trained models to store the knowledge into an enormous number of model parameters. The training of these models is composed of lots of dense algebras, requiring a huge amount of hardware resources. Recently, sparsely-gated Mixture-of-Experts (MoEs) are becoming more popular and have demonstrated impressive pretraining scalability in various downstream tasks. However, such a sparse conditional computation may not be effective as expected in practical systems due to the routing imbalance and fluctuation problems. Generally, MoEs are becoming a new data analytics paradigm in the data life cycle and suffering from unique challenges at scales, complexities, and granularities never before possible. In this paper, we propose a novel DNN training framework, FlexMoE, which systematically and transparently address the inefficiency caused by dynamic dataflow. We first present an empirical analysis on the problems and opportunities of training MoE models, which motivates us to overcome the routing imbalance and fluctuation problems by a dynamic expert management and device placement mechanism. Then we introduce a novel scheduling module over the existing DNN runtime to monitor the data flow, make the scheduling plans, and dynamically adjust the model-to-hardware mapping guided by the real-time data traffic. A simple but efficient heuristic algorithm is exploited to dynamically optimize the device placement during training. We have conducted experiments on both NLP models (e.g., BERT and GPT) and vision models (e.g., Swin). And results show FlexMoE can achieve superior performance compared with existing systems on real-world workloads -- FlexMoE outperforms DeepSpeed by 1.70x on average and up to 2.10x, and outperforms FasterMoE by 1.30x on average and up to 1.45x.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 27 of its 48 references that are in the map. First posted on arXiv 8 Apr 2023.

Builds on

27 of 48 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022StableMoE: Stable Routing Strategy for Mixture of Experts18 Apr
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  12. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
13 more
  1. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  2. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  4. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  5. 2021Hash Layers For Large Sparse Modelscited by 236
  6. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  7. 2021M6: A Chinese Multimodal Pretrainercited by 91
  8. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  9. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training Systemcited by 36
  10. 2021HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Frameworkcited by 28
  11. 2022Towards Communication-efficient Vertical Federated Learning Training via Cache-enabled Local Updatescited by 15
  12. 2023Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencentcited by 8
  13. 2019An Experimental Evaluation of Large Scale GBDT Systemscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 12; 2025: 18; 2026: 17.

52 papers.

Sort:

By area:

  1. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  2. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 83
  3. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  4. 2024KeyVideoLLM: Towards Large-scale Video Keyframe Selectioncited by 52
  5. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  6. 2024LocMoE: A Low-Overhead MoE for Large Language Model Trainingcited by 18
  7. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Modelscited by 17
  8. 2023Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsitycited by 14
  9. 2025FlashMoE: Fast Distributed MoE in a Single Kernelcited by 13
  10. 2023Experts Weights Averaging: A New General Training Scheme for Vision Transformerscited by 12
  11. 2025Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUscited by 12
  12. 2025GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inferencecited by 12
38 more
  1. 2024Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizingcited by 10
  2. 2024Synth-Empathy: Towards High-Quality Synthetic Empathy Datacited by 10
  3. 2025Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Schedulingcited by 9
  4. 2026UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancingcited by 8
  5. 2024PAS: Data-Efficient Plug-and-Play Prompt Augmentation Systemcited by 7
  6. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
  7. 2024Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Schedulingcited by 7
  8. 2025HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUscited by 7
  9. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placementcited by 7
  10. 2024SynthVLM: Towards High-Quality and Efficient Synthesis of Image-Caption Datasets for Vision-Language Modelscited by 6
  11. 2024EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inferencecited by 6
  12. 2025SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decouplingcited by 5
  13. 2025Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokenscited by 5
  14. 2026LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Trainingcited by 5
  15. 2026Grouter: Decoupling Routing from Representation for Accelerated MoE Trainingcited by 5
  16. 2025MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancingcited by 4
  17. 2025HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swapcited by 4
  18. 2024Sparse Checkpointing for Fast and Reliable MoE Trainingcited by 3
  19. 2025Optimizing Distributed Deployment of Mixture-of-Experts Model Inference in Serverless Computingcited by 3
  20. 2026ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learningcited by 2
  21. 2026Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-trainingcited by 2
  22. 2025iServe: An Intent-based Serving System for LLMscited by 1
  23. 2025Fine-grained MoE Load Balancing with Linear Programmingcited by 1
  24. 2025Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelismcited by 1
  25. 2026ReaLB: Real-Time Load Balancing for Multimodal MoE Inferencecited by 1
  26. 2026On the Design of Mixture-of-Experts for Dynamic Gaussian Splattingcited by 1
  27. 2024Efficient-Empathy: Towards Efficient and Effective Selection of Empathy Datacited by 0
  28. 2025You Don't Need All Attentions: Distributed Dynamic Fine-Tuning for Foundation Modelscited by 0
  29. 2025Dynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Expertscited by 0
  30. 2025Concertina: Data-Centric Adaptive Pipeline Parallelism for Efficient Heterogeneous Long-Context LLM Trainingcited by 0
  31. 2026MoE-Hub: Taming Software Complexity for Seamless MoE Overlap with Hardware-Accelerated Communication on Multi-GPU Systemscited by 0
  32. 2026Fast MoE Inference via Predictive Prefetching and Expert Replicationcited by 0
  33. 2026Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatorycited by 0
  34. 2026Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inferencecited by 0
  35. 2026Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placementcited by 0
  36. 2026Searching for Plans You Can Actually Build: A Realizability-Aware Full-Space Optimizer for MoE Training and Servingcited by 0
  37. 2026TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimescited by 0
  38. 2026Flattening Every Memory Peak in Long-Context Mixture-of-Experts Trainingcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts
  2. 2026DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
  3. 2025Load Balancing Mixture of Experts with Similarity Preserving Routers
  4. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing
  5. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts
  6. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts
  7. 2025DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs
  8. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts