mlsanityData to 7 Oct 2026

Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning

arXiv 2506.01656 ↗First posted 2 Jun 2025cs.LG stat.MLcited by 4

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Mixture of Experts (MoE), an ensemble of specialized models equipped with a router that dynamically distributes each input to appropriate experts, has achieved successful results in the field of machine learning. However, theoretical understanding of this architecture is falling behind due to its inherent complexity. In this paper, we theoretically study the sample and runtime complexity of MoE following the stochastic gradient descent (SGD) when learning a regression task with an underlying cluster structure of single index models. On the one hand, we prove that a vanilla neural network fails in detecting such a latent organization as it can only process the problem as a whole. This is intrinsically related to the concept of information exponent which is low for each cluster, but increases when we consider the entire task. On the other hand, we show that a MoE succeeds in dividing this problem into easier subproblems by leveraging the ability of each expert to weakly recover the simpler function corresponding to an individual cluster. To the best of our knowledge, this work is among the first to explore the benefits of the MoE framework by examining its SGD dynamics in the context of nonlinear regression.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 15 of its 60 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

15 of 60 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Mixture of experts models for multilevel data: modelling framework and approximation theory30 Sep

Other references · 14

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  3. 2024Mixtral of Expertscited by 1,785
  4. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  5. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  6. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Modelscited by 57
  7. 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 53
  8. 2024Theory on Mixture-of-Experts in Continual Learningcited by 50
  9. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Expertscited by 36
  10. 2024Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functionscited by 35
  11. 2023Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index modelscited by 25
  12. 2024Proactive Gradient Conflict Mitigation in Multi-Task Learning: A Sparse Training Perspectivecited by 12
2 more
  1. 2024Robust Feature Learning for Multi-Index Models in High Dimensionscited by 5
  2. 2024Gradient dynamics for low-rank fine-tuning beyond kernelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Expertscited by 1
  2. 2026Multi-Modal Time Series Prediction via Mixture of Modulated Expertscited by 1
  3. 2026Soft-to-Hard Routing in Sparse Mixture-of-Experts Modelscited by 1
  4. 2026Few-Shot Demonstrations Elicit the Use of In-Context World Representations in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Towards Understanding Mixture of Experts in Deep Learning
  2. 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
  3. 2023Revisiting Single-gated Mixtures of Experts
  4. 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks
  5. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
  6. 2024Learning Mixtures of Experts with EM: A Mirror Descent Perspective
  7. 2025Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms
  8. 2018Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithms