mlsanityData to 7 Oct 2026

Fine-grained MoE Load Balancing with Linear Programming

arXiv 2511.16947 ↗First posted 21 Nov 2025cs.DCcited by 1

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) has emerged as a promising approach to scale up deep learning models due to its significant reduction in computational resources. However, the dynamic nature of MoE leads to load imbalance among experts, severely impacting training efficiency. While previous research has attempted to address the load balancing challenge, existing solutions either compromise model accuracy or introduce additional system overhead. As a result, they fail to achieve fine-grained load balancing, which is crucial to optimizing training efficiency. We propose a novel parallelization strategy to achieve fine-grained load balancing in MoE systems. Our system is capable of achieving optimal load balancing in every micro-batch through efficient token scheduling across GPUs. Our experimental results demonstrate that MicroMoE improves the end-to-end training throughput by up to 47.6% compared with the state-of-the-art system, and almost consistently achieves optimal load balance among GPUs.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 41 of its 62 references that are in the map. First posted on arXiv 21 Nov 2025.

Builds on

41 of 62 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts27 Feb
  2. 2025Themis: Efficient Sparse Model Training Through Fully Sharded Sparse Data Parallelism4 Feb
  3. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models21 Jan
  4. 2024Pro-Prophet: A Systematic Load Balancing Method for Efficient Parallel Training of Large-scale MoE Models15 Nov
  5. 2024LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing13 Nov
  6. 2024Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing25 Apr
  7. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  8. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  9. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  10. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services20 May
  11. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan
  12. 2021FastMoE: A Fast Mixture-of-Expert Training System24 Mar

Other references · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
17 more
  1. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  2. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  3. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  4. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  5. 2021Hash Layers For Large Sparse Modelscited by 236
  6. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 191
  7. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  8. 2015Parallelizing the dual revised simplex methodcited by 83
  9. 2023Multi-modal Gated Mixture of Local-to-Global Experts for Dynamic Image Fusioncited by 54
  10. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 52
  11. 2023Zero Bubble Pipeline Parallelismcited by 39
  12. 2025Advancing Expert Specialization for Better MoEcited by 37
  13. 2024LocMoE: A Low-Overhead MoE for Large Language Model Trainingcited by 18
  14. 2024Yuan 2.0-M32: Mixture of Experts with Attention Routercited by 18
  15. 2025Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithmscited by 16
  16. 2023TA-MoE: Topology-Aware Large Scale Mixture-of-Expert Trainingcited by 15
  17. 2025Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsitycited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Routing-Free Mixture-of-Expertscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025MemFine: Memory-Aware Fine-Grained Scheduling for MoE Training
  2. 2026FEPLB: Exploiting Copy Engines for Nearly Free MoE Load Balancing in Distributed Training
  3. 2024Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation
  4. 2026Scalable Training of Mixture-of-Experts Models with Megatron Core
  5. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
  6. 2025MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing
  7. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing
  8. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training