mlsanityData to 7 Oct 2026

The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications

arXiv 2602.08019 ↗First posted 8 Feb 2026cs.LG cs.AIcited by 2

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

The sparse Mixture of Experts(MoE) architecture has evolved as a powerful approach for scaling deep learning models to more parameters with comparable computation cost. As an important branch of large language model(LLM), MoE model only activate a subset of experts based on a routing network. This sparse conditional computation mechanism significantly improves computational efficiency, paving a promising path for greater scalability and cost-efficiency. It not only enhance downstream applications such as natural language processing, computer vision, and multimodal in various horizontal domains, but also exhibit broad applicability across vertical domains. Despite the growing popularity and application of MoE models across various domains, there lacks a systematic exploration of recent advancements of MoE in many important fields. Existing surveys on MoE suffer from limitations such as lack coverage or none extensively exploration of key areas. This survey seeks to fill these gaps. In this paper, Firstly, we examine the foundational principles of MoE, with an in-depth exploration of its core components-the routing network and expert network. Subsequently, we extend beyond the centralized paradigm to the decentralized paradigm, which unlocks the immense untapped potential of decentralized infrastructure, enables democratization of MoE development for broader communities, and delivers greater scalability and cost-efficiency. Furthermore we focus on exploring its vertical domain applications. Finally, we also identify key challenges and promising future research directions. To the best of our knowledge, this survey is currently the most comprehensive review in the field of MoE. We aim for this article to serve as a valuable resource for both researchers and practitioners, enabling them to navigate and stay up-to-date with the latest advancements.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 90 of its 130 references that are in the map. First posted on arXiv 8 Feb 2026.

Builds on

90 of 130 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025LiMoE: Mixture of LiDAR Representation Learners from Automotive Scenes7 Jan
  2. 2022A Review of Sparse Expert Models in Deep Learning4 Sep
  3. 2022Sparsely-gated Mixture-of-Expert Layers for CNN Interpretability22 Apr
  4. 2021Scaling Vision with Sparse Mixture of Experts10 Jun

Other references · 86

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  12. 2024Mixtral of Expertscited by 1,785
38 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  3. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  5. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  7. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  8. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  9. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
  10. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  11. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  12. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  13. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  14. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  15. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  16. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  17. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  18. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  19. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  20. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 115
  21. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
  22. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  23. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  24. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Drivingcited by 85
  25. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  26. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  27. 2025ARTEMIS: Autoregressive End-to-End Trajectory Planning with Mixture of Experts for Autonomous Drivingcited by 66
  28. 2021Distributed Deep Learning in Open Collaborationscited by 58
  29. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 58
  30. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Modelscited by 57
  31. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Modelscited by 53
  32. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Trainingcited by 47
  33. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  34. 2020Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Expertscited by 43
  35. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Modelscited by 43
  36. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training Systemcited by 36
  37. 2022Safe Real-World Autonomous Driving by Learning to Predict and Plan with a Mixture of Expertscited by 28
  38. 2024HMoE: Heterogeneous Mixture of Experts for Language Modelingcited by 28

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026MoPET: Parameter-Efficient Mixture-of-Experts for Unified Medical Image Classificationcited by 0
  2. 2026MEOX: Compact Multimodal Mixture-of-Experts for Earth Observationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixture of Experts in Large Language Models
  2. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
  3. 2026Design and Behavior of Sparse Mixture-of-Experts Layers in CNN-based Semantic Segmentation
  4. 2026Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization
  5. 2026Towards a Statistical Understanding of Mixture-of-Experts
  6. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts
  7. 2023Scaling Vision-Language Models with Sparse Mixture of Experts
  8. 2024Routers in Vision Mixture of Experts: An Empirical Study