mlsanityData to 7 Oct 2026

Mixture of Experts (MoE): A Big Data Perspective

arXiv 2501.16352 ↗First posted 18 Jan 2025cs.LG cs.AIcited by 17

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

As the era of big data arrives, traditional artificial intelligence algorithms have difficulty processing the demands of massive and diverse data. Mixture of experts (MoE) has shown excellent performance and broad application prospects. This paper provides an in-depth review and analysis of the latest progress in this field from multiple perspectives, including the basic principles, algorithmic models, key technical challenges, and application practices of MoE. First, we introduce the basic concept of MoE and its core idea and elaborate on its advantages over traditional single models. Then, we discuss the basic architecture of MoE and its main components, including the gating network, expert networks, and learning algorithms. Next, we review the applications of MoE in addressing key technical issues in big data. For each challenge, we provide specific MoE solutions and their innovations. Furthermore, we summarize the typical use cases of MoE in various application domains. This fully demonstrates the powerful capability of MoE in big data processing. We also analyze the advantages of MoE in big data environments. Finally, we explore the future development trends of MoE. We believe that MoE will become an important paradigm of artificial intelligence in the era of big data. In summary, this paper systematically elaborates on the principles, techniques, and applications of MoE in big data processing, providing theoretical and practical references to further promote the application of MoE in real scenarios.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 79 of its 143 references that are in the map. First posted on arXiv 18 Jan 2025.

Builds on

79 of 143 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024HoME: Hierarchy of Multi-Gate Experts for Multi-Task Learning at Kuaishou10 Aug
  2. 2022Interpretable Mixture of Experts5 Jun
  3. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning7 Jun
  4. 2019Regularized Estimation and Feature Selection in Mixtures of Gaussian-Gated Experts Models12 Sep
  5. 2018Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithms21 Feb

Other references · 74

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  6. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  7. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  8. 2017Dynamic Routing Between Capsulescited by 689
  9. 2016Learning to learn by gradient descent by gradient descentcited by 616
  10. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  11. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  12. 2020A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospectscited by 396
38 more
  1. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  2. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  3. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  4. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  5. 2023Multimodal Large Language Models: A Surveycited by 193
  6. 2021Towards Understanding Knowledge Distillationcited by 158
  7. 2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Pathscited by 141
  8. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  9. 2023Large Language Models in Law: A Surveycited by 128
  10. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 110
  11. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 101
  12. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
  13. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
  14. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 64
  15. 2022Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensemblescited by 59
  16. 2024MoA: Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generationcited by 51
  17. 2022Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-Expertscited by 50
  18. 2024Large Language Models for Medicine: A Surveycited by 38
  19. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Servicescited by 37
  20. 2023Merging Experts into One: Improving Computational Efficiency of Mixture of Expertscited by 36
  21. 2023Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networkscited by 35
  22. 2024Customizing Language Models with Instance-wise LoRA for Sequential Recommendationcited by 32
  23. 2022Text2Human: Text-Driven Controllable Human Image Generationcited by 31
  24. 2024AUFormer: Vision Transformers are Parameter-Efficient Facial Action Unit Detectorscited by 29
  25. 2024M4oE: A Foundation Model for Medical Multimodal Image Segmentation with Mixture of Expertscited by 26
  26. 2021Specializing Versatile Skill Libraries using Local Mixture of Expertscited by 24
  27. 2023Model-as-a-Service (MaaS): A Surveycited by 24
  28. 2023Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Expertscited by 23
  29. 2021RetGen: A Joint framework for Retrieval and Grounded Text Generation Modelingcited by 22
  30. 2024Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlappingcited by 22
  31. 2023Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perceptioncited by 19
  32. 2024Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Expertscited by 19
  33. 2020Continuous Action Reinforcement Learning from a Mixture of Interpretable Expertscited by 17
  34. 2022Robust Hybrid Learning With Expert Augmentationcited by 17
  35. 2022Redesigning Multi-Scale Neural Network for Crowd Countingcited by 17
  36. 2023Data Scarcity in Recommendation Systems: A Surveycited by 17
  37. 2023Adaptive Gating in Mixture-of-Experts based Language Modelscited by 16
  38. 2024MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routingcited by 16

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 7.

17 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  2. 2025Mixture of Experts in Large Language Modelscited by 15
  3. 2025Unraveling the Localized Latents: Learning Stratified Manifold Structures in LLM Embedding Space with Sparse Mixture-of-Expertscited by 7
  4. 2025Evaluation of LLMs for mathematical problem solvingcited by 7
  5. 2025Thinking Beyond Tokens: From Brain-Inspired Intelligence to Cognitive Foundations for Artificial General Intelligence and its Societal Impactcited by 6
  6. 2025DynaMo: Runtime Switchable Quantization for MoE with Cross-Dataset Adaptationcited by 4
  7. 2025DE3S: Dual-Enhanced Soft-Sparse-Shape Learning for Medical Early Time-Series Classificationcited by 2
  8. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applicationscited by 2
  9. 2025Advances in Large Language Models for Medicinecited by 1
  10. 2025Graph-of-Causal Evolution: Challenging Chain-of-Model for Reasoningcited by 0
  11. 2025History-Augmented Contrastive Learning With Soft Mixture of Experts for Blind Super-Resolution of Planetary Remote Sensing Imagescited by 0
  12. 2026Let Experts Feel Uncertainty: A Multi-Expert Label Distribution Approach to Probabilistic Time Series Forecastingcited by 0
5 more
  1. 2026Robust Global-Local Behavior Arbitration via Continuous Command Fusion Under LiDAR Errorscited by 0
  2. 2026Mixture-of-Experts in Remote Sensing: A Surveycited by 0
  3. 2026A novel YOLO26-MoE optimized by an LLM agent for insulator fault detection considering UAV imagescited by 0
  4. 2026Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Expertscited by 0
  5. 2026Generalist-Specialist Mixture-of-Experts for Rare Pathology Detection in Multimodal Imagingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Models
  2. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
  3. 2017An Introduction to the Practical and Theoretical Aspects of Mixture-of-Experts Modeling
  4. 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
  5. 2023Revisiting Single-gated Mixtures of Experts
  6. 2022MoEC: Mixture of Expert Clusters
  7. 2023Improving Expert Specialization in Mixture of Experts
  8. 2024DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models