mlsanityData to 7 Oct 2026

A Survey on Mixture of Experts in Large Language Models

arXiv 2407.06204 ↗First posted 26 Jun 2024cs.LG cs.CLcited by 233

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Large language models (LLMs) have garnered unprecedented advancements across diverse fields, ranging from natural language processing to computer vision and beyond. The prowess of LLMs is underpinned by their substantial model size, extensive and diverse datasets, and the vast computational power harnessed during training, all of which contribute to the emergent abilities of LLMs (e.g., in-context learning) that are not present in small models. Within this context, the mixture of experts (MoE) has emerged as an effective method for substantially scaling up model capacity with minimal computation overhead, gaining significant attention from academia and industry. Despite its growing prevalence, there lacks a systematic and comprehensive review of the literature on MoE. This survey seeks to bridge that gap, serving as an essential resource for researchers delving into the intricacies of MoE. We first briefly introduce the structure of the MoE layer, followed by proposing a new taxonomy of MoE. Next, we overview the core designs for various MoE models including both algorithmic and systemic aspects, alongside collections of available open-source implementations, hyperparameter configurations and empirical evaluations. Furthermore, we delineate the multifaceted applications of MoE in practice, and outline some potential directions for future research. To facilitate ongoing updates and the sharing of cutting-edge advances in MoE research, we have established a resource repository at https://github.com/withinmiaov/A-Survey-on-Mixture-of-Experts-in-LLMs.

Key facts. Cited by 233 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 98 of its 220 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

98 of 220 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques4 Jun
  2. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models3 Jun
  3. 2024Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training6 May
  4. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollars11 Apr
  5. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models8 Apr
  6. 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
  7. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM12 Mar
  8. 2024HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts20 Feb
  9. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models29 Jan
  10. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  11. 2024Mixtral of Experts8 Jan
  12. 2023ModuleFormer: Modularity Emerges from Mixture-of-Experts7 Jun
7 more
  1. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models24 May
  2. 2022No Language Left Behind: Scaling Human-Centered Machine Translation11 Jul
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  4. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  5. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models22 Sep
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 79

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  11. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  12. 2023Mistral 7Bcited by 3,065
38 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2020GLU Variants Improve Transformercited by 1,724
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2024A Survey on Large Language Models for Code Generationcited by 646
  9. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  10. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  11. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  12. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  13. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  14. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  15. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  16. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  17. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  18. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  19. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  20. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  21. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  22. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  23. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 125
  24. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 125
  25. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  26. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 110
  27. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 110
  28. 2023Skywork: A More Open Bilingual Foundation Modelcited by 110
  29. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
  30. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  31. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 101
  32. 2024Higher Layers Need More LoRA Expertscited by 101
  33. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  34. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  35. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  36. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82
  37. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 69
  38. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 65

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 116; 2026: 86.

233 papers.

Sort:

By area:

  1. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  2. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  3. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  4. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Drivingcited by 85
  5. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  6. 2024Attention Heads of Large Language Models: A Surveycited by 65
  7. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 39
  8. 2025Advancing Expert Specialization for Better MoEcited by 37
  9. 2025Delta Decompression for MoE-based LLMs Compressioncited by 32
  10. 2024LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Acceleratorscited by 31
  11. 2024DeMo: Decoupled Feature-Based Mixture of Experts for Multi-Modal Object Re-Identificationcited by 29
  12. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routingcited by 26
38 more
  1. 2025Towards Understanding the Safety Boundaries of DeepSeek Models: Evaluation and Findingscited by 24
  2. 2025DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruningcited by 24
  3. 2025Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initializationcited by 23
  4. 2025Steering MoE LLMs via Expert (De)Activationcited by 23
  5. 2025Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidancecited by 23
  6. 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 22
  7. 2024Multi-Expert Adaptive Selection: Task-Balancing for All-in-One Image Restorationcited by 18
  8. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 18
  9. 2025The Rise of Small Language Models in Healthcare: A Comprehensive Surveycited by 18
  10. 2026Intelligent AI Delegationcited by 16
  11. 2024DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Modelscited by 15
  12. 2025Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyondcited by 15
  13. 2025Scaling Law for Quantization-Aware Trainingcited by 15
  14. 2025Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsitycited by 15
  15. 2024Leveraging Mixture of Experts for Improved Speech Deepfake Detectioncited by 14
  16. 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inferencecited by 14
  17. 2025SoK: Large Language Model Copyright Auditing via Fingerprintingcited by 14
  18. 2024COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesiscited by 13
  19. 2024γ-MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large Language Modelscited by 13
  20. 2026Heterogeneous Agent Collaborative Reinforcement Learningcited by 13
  21. 2025Scaling Laws for Upcycling Mixture-of-Experts Language Modelscited by 12
  22. 2024Stealing User Prompts from Mixture of Expertscited by 10
  23. 2025DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanismcited by 10
  24. 2025Ensembles of Low-Rank Expert Adapterscited by 9
  25. 2025Safety Evaluation of DeepSeek Models in Chinese Contextscited by 9
  26. 2025Comparative Analysis Based on DeepSeek, ChatGPT, and Google Gemini: Features, Techniques, Performance, Future Prospectscited by 9
  27. 2025EAQuant: Enhancing Post-Training Quantization for MoE Models via Expert-Aware Optimizationcited by 9
  28. 2025MoE-DP: An MoE-Enhanced Diffusion Policy for Robust Long-Horizon Robotic Manipulation with Skill Decomposition and Failure Recoverycited by 9
  29. 2025MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigmcited by 9
  30. 2024KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Modelscited by 8
  31. 2025Physics of Skill Learningcited by 8
  32. 2025On the Failure of Latent State Persistence in Large Language Modelscited by 8
  33. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  34. 2025S-DAG: A Subject-Based Directed Acyclic Graph for Multi-Agent Heterogeneous Reasoningcited by 8
  35. 2025Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Modelscited by 8
  36. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
  37. 2024Faster Language Models with Better Multi-Token Prediction Using Tensor Decompositioncited by 7
  38. 2025GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanismcited by 7

50 most cited of 233 All 233 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixture of Experts in Large Language Models
  2. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  3. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  4. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
  5. 2024Scaling Laws for Fine-Grained Mixture of Experts
  6. 2024Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models
  7. 2025FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models
  8. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models