mlsanityData to 7 Oct 2026

A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications

arXiv 2503.07137 ↗First posted 10 Mar 2025cs.LG cs.AIcited by 91

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Artificial intelligence (AI) has achieved astonishing successes in many domains, especially with the recent breakthroughs in the development of foundational large models. These large models, leveraging their extensive training data, provide versatile solutions for a wide range of downstream tasks. However, as modern datasets become increasingly diverse and complex, the development of large AI models faces two major challenges: (1) the enormous consumption of computational resources and deployment difficulties, and (2) the difficulty in fitting heterogeneous and complex data, which limits the usability of the models. Mixture of Experts (MoE) models has recently attracted much attention in addressing these challenges, by dynamically selecting and activating the most relevant sub-models to process input data. It has been shown that MoEs can significantly improve model performance and efficiency with fewer resources, particularly excelling in handling large-scale, multimodal data. Given the tremendous potential MoE has demonstrated across various domains, it is urgent to provide a comprehensive summary of recent advancements of MoEs in many important fields. Existing surveys on MoE have their limitations, e.g., being outdated or lacking discussion on certain key areas, and we aim to address these gaps. In this paper, we first introduce the basic design of MoE, including gating functions, expert networks, routing mechanisms, training strategies, and system design. We then explore the algorithm design of MoE in important machine learning paradigms such as continual learning, meta-learning, multi-task learning, and reinforcement learning. Additionally, we summarize theoretical studies aimed at understanding MoE and review its applications in computer vision and natural language processing. Finally, we discuss promising future research directions.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 114 of its 305 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

114 of 305 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Towards Foundational Models for Dynamical System Reconstruction: Hierarchical Meta-Learning via Mixture of Experts7 Feb
  2. 2025Mixture of Experts (MoE): A Big Data Perspective18 Jan
  3. 2024On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions3 Oct
  4. 2024Decomposing the Neurons: Activation Sparsity via Mixture of Experts for Continual Test Time Adaptation26 May
  5. 2024On Least Square Estimation in Softmax Gating Mixture of Experts5 Feb
  6. 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts22 Oct
  7. 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts25 Sep
  8. 2023Revisiting Single-gated Mixtures of Experts11 Apr

Other references · 106

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  11. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
38 more
  1. 2024Mixtral of Expertscited by 1,785
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  4. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  6. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  8. 2024DeepSeek-V3 Technical Reportcited by 724
  9. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  10. 2016Neural Architectures for Named Entity Recognitioncited by 548
  11. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  12. 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
  13. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  14. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  15. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  16. 2018Neural Processescited by 307
  17. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 280
  18. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  19. 2023From Sparse to Soft Mixtures of Expertscited by 245
  20. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  21. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  22. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  23. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  24. 2024Continual Learning with Pre-Trained Models: A Surveycited by 151
  25. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
  26. 2020A Neural Dirichlet Process Mixture Model for Task-Free Continual Learningcited by 133
  27. 2019One-Shot Federated Learningcited by 124
  28. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 118
  29. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
  30. 2021FedGNN: Federated Graph Neural Network for Privacy-Preserving Recommendationcited by 97
  31. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  32. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 89
  33. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  34. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  35. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 79
  36. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  37. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Modelscited by 71
  38. 2018Multi-Source Domain Adaptation with Mixture of Expertscited by 62

50 most cited of 106 All 106 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 36; 2026: 55.

91 papers.

Sort:

By area:

  1. 2026Advancing Open-source World Modelscited by 144
  2. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 71
  3. 2025Mixture of Experts in Large Language Modelscited by 15
  4. 2025Unveiling Super Experts in Mixture-of-Experts Large Language Modelscited by 15
  5. 2025MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Mergingcited by 14
  6. 2025Optimus-3: Dual-Router Aligned Mixture-of-Experts Agent with Dual-Granularity Reasoning-Aware Policy Optimizationcited by 12
  7. 2026Steer2Adapt: Dynamically Composing Steering Vectors Elicits Efficient Adaptation of LLMscited by 7
  8. 2025Can AI Truly Represent Your Voice in Deliberations? A Comprehensive Study of Large-Scale Opinion Aggregation with LLMscited by 6
  9. 2025Unbiased Gradient Low-Rank Projectioncited by 6
  10. 2026Sparse Models, Sparse Safety: Unsafe Routes in Mixture-of-Experts LLMscited by 6
  11. 2025GEMINUS: Dual-aware Global and Scene-Adaptive Mixture-of-Experts for End-to-End Autonomous Drivingcited by 5
  12. 2025Mixture-of-Shape-Experts (MoSE): End-to-End Shape Dictionary Framework to Prompt SAM for Generalizable Medical Segmentationcited by 4
38 more
  1. 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Taskscited by 4
  2. 2025The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMMcited by 4
  3. 2026A Reasoning-Enabled Vision-Language Foundation Model for Chest X-ray Interpretationcited by 4
  4. 2026φ-Balancing for Mixture-of-Experts Trainingcited by 4
  5. 2026Variational Inference, Entropy, and Orthogonality: A Unified Theory of Mixture-of-Expertscited by 3
  6. 2026Sparsity is Combinatorial Depth: Quantifying MoE Expressivity via Tropical Geometrycited by 3
  7. 2025Knowing When to Quit: Probabilistic Early Exits for Speech Separationcited by 2
  8. 2025A Mixture of Experts Gating Network for Enhanced Surrogate Modeling in External Aerodynamicscited by 2
  9. 2025One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learningcited by 2
  10. 2025REN: Anatomically-Informed Mixture-of-Experts for Interstitial Lung Disease Diagnosiscited by 2
  11. 2025AB-PINNs: Adaptive-Basis Physics-Informed Neural Networks for Residual-Driven Domain Decompositioncited by 2
  12. 2025Sparsity and Superposition in Mixture of Expertscited by 2
  13. 2025Metacognitive Sensitivity for Test-Time Dynamic Model Selectioncited by 2
  14. 2025RepetitionCurse: Measuring and Understanding Router Imbalance in Mixture-of-Experts LLMs under DoS Stresscited by 2
  15. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applicationscited by 2
  16. 2026Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLMcited by 2
  17. 2026Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-trainingcited by 2
  18. 2025EvidenceMoE: A Physics-Guided Mixture-of-Experts with Evidential Critics for Advancing Fluorescence Light Detection and Ranging in Scattering Mediacited by 1
  19. 2025Second Opinion Matters: Towards Adaptive Clinical AI via the Consensus of Expert Model Ensemblecited by 1
  20. 2025CAT: Curvature-Adaptive Transformers for Geometry-Aware Learningcited by 1
  21. 2025Sparse Query Attention (SQA): A Computationally Efficient Attention Mechanism with Query Heads Reductioncited by 1
  22. 2025Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Expertscited by 1
  23. 2025YOLO Meets Mixture-of-Experts: Adaptive Expert Routing for Robust Object Detectioncited by 1
  24. 2025EBind: a practical approach to space bindingcited by 1
  25. 2025Divided We Fall: Defending Against Adversarial Attacks via Soft-Gated Fractional Mixture-of-Experts with Randomized Adversarial Trainingcited by 1
  26. 2026Revisiting the Shape Convention of Transformer Language Modelscited by 1
  27. 2026Beyond Student: An Asymmetric Network for Neural Network Inheritancecited by 1
  28. 2026CoGR-MoE: Concept-Guided Expert Routing with Consistent Selection and Flexible Reasoning for Visual Question Answeringcited by 1
  29. 2026Bridging the Detection-to-Abstention Gap in Reasoning Models under Insufficient Informationcited by 1
  30. 2025Hybrid Real- and Complex-valued Neural Network Architecturecited by 0
  31. 2025Learning to Route: Per-Sample Adaptive Routing for Multimodal Multitask Predictioncited by 0
  32. 2025Stochastic activationscited by 0
  33. 2025Mixture-of-Transformers Learn Faster: A Theoretical Study on Classification Problemscited by 0
  34. 2025History-Augmented Contrastive Learning With Soft Mixture of Experts for Blind Super-Resolution of Planetary Remote Sensing Imagescited by 0
  35. 2025MixtureKit: A General Framework for Composing, Training, and Visualizing Mixture-of-Experts Modelscited by 0
  36. 2025MMCTOP: A Multimodal Textualization and Mixture-of-Experts Framework for Clinical Trial Outcome Predictioncited by 0
  37. 2025Hybrid Quantum-Classical Mixture of Experts: Unlocking Topological Advantage via Interference-Based Routingcited by 0
  38. 2025GLUE: Gradient-free Learning to Unify Expertscited by 0

50 most cited of 91 All 91 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2017An Introduction to the Practical and Theoretical Aspects of Mixture-of-Experts Modeling
  2. 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
  3. 2025Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms
  4. 2025Towards a Comprehensive Scaling Law of Mixture-of-Experts
  5. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
  6. 2025Rethinking Sparse Mixture of Experts from a Unified Perspective
  7. 2024Explainable data-driven modeling via mixture of experts: towards effective blending of grey and black-box models
  8. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts