mlsanityData to 14 Oct 2026

Mixture of Experts in Large Language Models

arXiv 2507.11181 ↗First posted 15 Jul 2025cs.LG cs.AIcited by 15

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

This paper presents a comprehensive review of the Mixture-of-Experts (MoE) architecture in large language models, highlighting its ability to significantly enhance model performance while maintaining minimal computational overhead. Through a systematic analysis spanning theoretical foundations, core architectural designs, and large language model (LLM) applications, we examine expert gating and routing mechanisms, hierarchical and sparse MoE configurations, meta-learning approaches, multimodal and multitask learning scenarios, real-world deployment cases, and recent advances and challenges in deep learning. Our analysis identifies key advantages of MoE, including superior model capacity compared to equivalent Bayesian approaches, improved task-specific performance, and the ability to scale model capacity efficiently. We also underscore the importance of ensuring expert diversity, accurate calibration, and reliable inference aggregation, as these are essential for maximizing the effectiveness of MoE architectures. Finally, this review outlines current research limitations, open challenges, and promising future directions, providing a foundation for continued innovation in MoE architecture and its applications.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 77 of its 104 references that are in the map. First posted on arXiv 15 Jul 2025.

Builds on

77 of 104 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs3 Feb
  2. 2024Flexible and Effective Mixing of Large Language Models into a Mixture of Domain Experts30 Aug
  3. 2024Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Experts28 Aug
  4. 2024HMoE: Heterogeneous Mixture of Experts for Language Modeling20 Aug
  5. 2024Multi-Head Mixture-of-Experts23 Apr
  6. 2024HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts20 Feb
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  8. 2024Mixtral of Experts8 Jan
  9. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer15 Oct
  10. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  11. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
2 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 63

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020End-to-End Object Detection with Transformerscited by 7,414
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  4. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2024Qwen2 Technical Reportcited by 2,120
  7. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  8. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  9. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  10. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  11. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  12. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
38 more
  1. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 203
  2. 2019MLPerf Inference Benchmarkcited by 199
  3. 2015Conditional Computation in Neural Networks for faster modelscited by 167
  4. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 150
  5. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  6. 2022Towards Universal Sequence Representation Learning for Recommender Systemscited by 121
  7. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 104
  8. 2024T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergycited by 94
  9. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  10. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 83
  11. 2024Complexity Experts are Task-Discriminative Learners for Any Image Restorationcited by 77
  12. 2022Towards Understanding Mixture of Experts in Deep Learningcited by 52
  13. 2022Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-Expertscited by 50
  14. 2024MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Modelscited by 46
  15. 2024M3oE: Multi-Domain Multi-Task Mixture-of Experts Recommendation Frameworkcited by 32
  16. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficientcited by 31
  17. 2025I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Expertscited by 30
  18. 2023Enhancing NeRF akin to Enhancing LLMs: Generalizable NeRF Transformer with Mixture-of-View-Expertscited by 27
  19. 2023Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Expertscited by 26
  20. 2022One Student Knows All Experts Know: From Sparse to Densecited by 25
  21. 2024Routers in Vision Mixture of Experts: An Empirical Studycited by 23
  22. 2025Load Balancing Mixture of Experts with Similarity Preserving Routerscited by 22
  23. 2024BiMediX: Bilingual Medical Mixture of Experts LLMcited by 21
  24. 2024Towards an empirical understanding of MoE design choicescited by 20
  25. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 18
  26. 2023Improving Expert Specialization in Mixture of Expertscited by 17
  27. 2023Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Expertscited by 17
  28. 2023MoCaE: Mixture of Calibrated Experts Significantly Improves Object Detectioncited by 17
  29. 2024Low-Rank Mixture-of-Experts for Continual Medical Image Segmentationcited by 16
  30. 2024On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functionscited by 16
  31. 2025OMoE: Diversifying Mixture of Low-Rank Adaptation by Orthogonal Finetuningcited by 15
  32. 2024MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Expertscited by 12
  33. 2024Learning More Generalized Experts by Merging Experts in Mixture-of-Expertscited by 12
  34. 2023Syn-Mediverse: A Multimodal Synthetic Dataset for Intelligent Scene Understanding of Healthcare Facilitiescited by 11
  35. 2024Mixture of Experts in Image Classification: What's the Sweet Spot?cited by 11
  36. 2025From Screens to Scenes: A Survey of Embodied AI in Healthcarecited by 10
  37. 2025MedMoE: Modality-Specialized Mixture of Experts for Medical Vision-Language Understandingcited by 10
  38. 2022HMOE: Hypernetwork-based Mixture of Experts for Domain Generalizationcited by 8

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 13.

15 papers.

Sort:

By area:

  1. 2026SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Modelscited by 13
  2. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Expertscited by 11
  3. 2026φ-Balancing for Mixture-of-Experts Trainingcited by 5
  4. 2025Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language Modelscited by 4
  5. 2026RASA: Routing-Aware Safety Alignment for Mixture-of-Experts Modelscited by 3
  6. 2026SD-MoE: Spectral Decomposition for Effective Expert Specializationcited by 3
  7. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Expertscited by 2
  8. 2026SPA-Cache: Singular Proxies for Adaptive Caching in Diffusion Language Modelscited by 1
  9. 2026Offline Semantic Guidance for Efficient Vision-Language-Action Policy Distillationcited by 1
  10. 2026RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanismcited by 1
  11. 2026State-of-the-Art Arabic Language Modeling with Sparse MoE Fine-Tuning and Chain-of-Thought Distillationcited by 0
  12. 2026Grad Detect: Gradient-Based Hallucination Detection in LLMscited by 0
3 more
  1. 2026DAIN: Dynamic Agent-Based Interaction Network for Efficient and Collaborative Multimodal Reasoningcited by 0
  2. 2026Task-Aware Federated Fine-Tuning for MoE-based Large Language Modelscited by 0
  3. 2026DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Compositioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  2. 2024A Survey on Mixture of Experts in Large Language Models
  3. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  4. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models
  5. 2025Advancing Expert Specialization for Better MoE
  6. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
  7. 2026Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints
  8. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models