mlsanityData to 7 Oct 2026

Towards Understanding Mixture of Experts in Deep Learning

arXiv 2208.02813 ↗First posted 4 Aug 2022cs.LG cs.AI stat.MLcited by 52

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

The Mixture-of-Experts (MoE) layer, a sparsely-activated model controlled by a router, has achieved great success in deep learning. However, the understanding of such architecture remains elusive. In this paper, we formally study how the MoE layer improves the performance of neural network learning and why the mixture model will not collapse into a single model. Our empirical results suggest that the cluster structure of the underlying problem and the non-linearity of the expert are pivotal to the success of MoE. To further understand this, we consider a challenging classification problem with intrinsic cluster structures, which is hard to learn using a single expert. Yet with the MoE layer, by choosing the experts as two-layer nonlinear convolutional neural networks (CNNs), we show that the problem can be learned successfully. Furthermore, our theory shows that the router can learn the cluster-center features, which helps divide the input complex problem into simpler linear classification sub-problems that individual experts can conquer. To our knowledge, this is the first result towards formally understanding the mechanism of the MoE layer for deep learning.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 27 of its 58 references that are in the map. First posted on arXiv 4 Aug 2022.

Builds on

27 of 58 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  8. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  9. 2016Language Modeling with Gated Convolutional Networkscited by 688
  10. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  11. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  12. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
15 more
  1. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  2. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  3. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  4. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  5. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  6. 2021Hash Layers For Large Sparse Modelscited by 236
  7. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  8. 2014Statistical guarantees for the EM algorithm: From population to sample-based analysiscited by 131
  9. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  10. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  11. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  12. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  13. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  14. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 48
  15. 2013Spectral Experts for Estimating Mixtures of Linear Regressionscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 16; 2024: 17; 2025: 12; 2026: 6.

52 papers.

Sort:

By area:

  1. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  2. 2022Vision Transformers provably learn spatial structurecited by 92
  3. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  4. 2023MSGNet: Learning Multi-Scale Inter-Series Correlations for Multivariate Time Series Forecastingcited by 50
  5. 2023Solving Regularized Exp, Cosh and Sinh Regression Problemscited by 44
  6. 2023Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networkscited by 35
  7. 2024AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agentscited by 32
  8. 2023Text Promptable Surgical Instrument Segmentation with Vision-Language Modelscited by 27
  9. 2024Diffusion Soup: Model Merging for Text-to-Image Diffusion Modelscited by 25
  10. 2023DAMEX: Dataset-aware Mixture-of-Experts for visual understanding of mixture-of-datasetscited by 23
  11. 2024Mixture of Parrots: Experts improve memorization more than reasoningcited by 20
  12. 2023The Benefits of Mixup for Feature Learningcited by 19
38 more
  1. 2024Node-wise Filtering in Graph Neural Networks: A Mixture of Experts Approachcited by 17
  2. 2023SQLPrompt: In-Context Text-to-SQL with Minimal Labeled Datacited by 16
  3. 2025VFX Creator: Animated Visual Effect Generation with Controllable Diffusion Transformercited by 15
  4. 2025Mixture of Experts in Large Language Modelscited by 15
  5. 2023Alternating Updates for Efficient Transformerscited by 13
  6. 2024Mixture of Experts in Image Classification: What's the Sweet Spot?cited by 11
  7. 2024Mixture of Link Predictors on Graphscited by 10
  8. 2023On the Clean Generalization and Robust Overfitting in Adversarial Training from Two Theoretical Views: Representation Complexity and Training Dynamicscited by 5
  9. 2024Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samplescited by 5
  10. 2025NeuroMoE: A Transformer-Based Mixture-of-Experts Framework for Multi-Modal Neurological Disorder Classificationcited by 5
  11. 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Studycited by 4
  12. 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Taskscited by 4
  13. 2025How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?cited by 4
  14. 2023Improving Transformer Performance for French Clinical Notes Classification Using Mixture of Experts on a Limited Datasetcited by 3
  15. 2023MoE-AMC: Enhancing Automatic Modulation Classification Performance Using Mixture-of-Expertscited by 3
  16. 2025One Model for All Tasks: Leveraging Efficient World Models in Multi-Task Planningcited by 3
  17. 2023ReactionTeam: Teaming Experts for Divergent Thinking Beyond Typical Reaction Patternscited by 2
  18. 2024Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsitycited by 2
  19. 2024Learning Mixtures of Experts with EM: A Mirror Descent Perspectivecited by 2
  20. 2025One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learningcited by 2
  21. 2026Geometric Regularization in Mixture-of-Experts: The Disconnect Between Weights and Activationscited by 2
  22. 2023Gated Self-supervised Learning For Improving Supervised Learningcited by 1
  23. 2023Simple Mechanisms for Representing, Indexing and Manipulating Conceptscited by 1
  24. 2024Exploring Domain Robust Lightweight Reward Models based on Router Mechanismcited by 1
  25. 2024Collaborative and Efficient Personalization with Mixtures of Adaptorscited by 1
  26. 2024Context-Aware Token Selection and Packing for Enhanced Vision Transformercited by 1
  27. 2024Steps are all you need: Rethinking STEM Education with Prompt Engineeringcited by 1
  28. 2025ECG-Soup: Harnessing Multi-Layer Synergy for ECG Foundation Modelscited by 1
  29. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Expertscited by 1
  30. 2023The Power of External Memory in Increasing Predictive Model Capacitycited by 0
  31. 2023Additive Class Distinction Maps using Branched-GANscited by 0
  32. 2024Predicting Exoplanetary Features with a Residual Model for Uniform and Gaussian Distributionscited by 0
  33. 2024Extracting thin film structures of energy materials using transformerscited by 0
  34. 2025NVR: Vector Runahead on NPUs for Sparse Memory Accesscited by 0
  35. 2025Mixture of Experts-augmented Deep Unfolding for Activity Detection in IRS-aided Systemscited by 0
  36. 2026Breaking the Bottlenecks: Scalable Diffusion Models for 3D Molecular Generationcited by 0
  37. 2026ExFusion: Efficient Transformer Training via Multi-Experts Fusioncited by 0
  38. 2026Improving Ensemble Forecasts of Abnormally Deflecting Tropical Cyclones with Fused Atmosphere-Ocean-Terrain Datacited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Robust Mixture-of-Expert Training for Convolutional Neural Networks
  2. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  3. 2018Deep Mixture of Experts via Shallow Embedding
  4. 2026Design and Behavior of Sparse Mixture-of-Experts Layers in CNN-based Semantic Segmentation
  5. 2025Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks
  6. 2025Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning
  7. 2026Mixture of Experts with Soft Nearest Neighbor Loss: Resolving Expert Collapse via Representation Disentanglement
  8. 2023Revisiting Single-gated Mixtures of Experts