mlsanityData to 14 Oct 2026

On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks

arXiv 2505.24205 ↗First posted 30 May 2025cs.LG stat.MLcited by 4

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Mixture-of-experts networks (MoEs) have demonstrated remarkable efficiency in modern deep learning. Despite their empirical success, the theoretical foundations underlying their ability to model complex tasks remain poorly understood. In this work, we conduct a systematic study of the expressive power of MoEs in modeling complex tasks with two common structural priors: low-dimensionality and sparsity. For shallow MoEs, we prove that they can efficiently approximate functions supported on low-dimensional manifolds, overcoming the curse of dimensionality. For deep MoEs, we show that O(L)-layer MoEs with E experts per layer can approximate piecewise functions comprising EL pieces with compositional sparsity, i.e., they can exhibit an exponential number of structured tasks. Our analysis reveals the roles of critical architectural components and hyperparameters in MoEs, including the gating mechanism, expert networks, the number of experts, and the number of layers, and offers natural suggestions for MoE variants.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 21 of its 46 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

21 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications10 Mar

Other references · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2024Qwen2 Technical Reportcited by 2,120
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  8. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  9. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  10. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 113
  11. 2016Deep vs. shallow networks : An approximation theory perspectivecited by 63
  12. 2015Provable approximation properties for deep neural networkscited by 60
8 more
  1. 2024Theory on Mixture-of-Experts in Continual Learningcited by 54
  2. 2022Towards Understanding Mixture of Experts in Deep Learningcited by 52
  3. 2024Mixture of Experts Meets Prompt-Based Continual Learningcited by 43
  4. 2023Deep Network Approximation: Beyond ReLU to Diverse Activation Functionscited by 33
  5. 2024Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modelingcited by 25
  6. 2022Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformerscited by 22
  7. 2023Rates of Approximation by ReLU Shallow Neural Networkscited by 12
  8. 2022A Theoretical View on Sparsely Activated Networkscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Expertscited by 2
  2. 2026Robustness of Mixtures of Experts to Feature Noisecited by 1
  3. 2026Soft-to-Hard Routing in Sparse Mixture-of-Experts Modelscited by 1
  4. 2026Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Functioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design
  2. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
  3. 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts
  4. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  5. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
  6. 2026Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization
  7. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts
  8. 2026MoE Lens -- An Expert Is All You Need