mlsanityData to 7 Oct 2026

Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Experts

arXiv 2510.07205 ↗First posted 8 Oct 2025cs.LG math.OCcited by 1

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Mixture-of-Experts (MoE) architectures have emerged as a cornerstone of modern AI systems. In particular, MoEs route inputs dynamically to specialized experts whose outputs are aggregated through weighted summation. Despite their widespread application, theoretical understanding of MoE training dynamics remains limited to either separate expert-router optimization or only top-1 routing scenarios with carefully constructed datasets. This paper advances MoE theory by providing convergence guarantees for joint training of soft-routed MoE models with non-linear routers and experts in a student-teacher framework. We prove that, with moderate over-parameterization, the student network undergoes a feature learning phase, where the router's learning process is ``guided'' by the experts, that recovers the teacher's parameters. Moreover, we show that a post-training pruning can effectively eliminate redundant neurons, followed by a provably convergent fine-tuning process that reaches global optimality. To our knowledge, our analysis is the first to bring novel insights in understanding the optimization landscape of the MoE architecture.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 39 of its 44 references that are in the map. First posted on arXiv 8 Oct 2025.

Builds on

39 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networks7 Jun

Other references · 38

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2024Mixtral of Expertscited by 1,785
  4. 2024DeepSeek-V3 Technical Reportcited by 724
  5. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  6. 2024Automated Design of Agentic Systemscited by 360
  7. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  8. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  9. 2024G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networkscited by 110
  10. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  11. 2017Learning One-hidden-layer Neural Networks with Landscape Designcited by 79
  12. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
26 more
  1. 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 53
  2. 2022Towards Understanding Mixture of Experts in Deep Learningcited by 52
  3. 2024Theory on Mixture-of-Experts in Continual Learningcited by 50
  4. 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
  5. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 45
  6. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Expertscited by 36
  7. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  8. 2025Emergence and scaling laws in SGD learning of shallow neural networkscited by 29
  9. 2025Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMscited by 24
  10. 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Expertscited by 23
  11. 2024On Least Square Estimation in Softmax Gating Mixture of Expertscited by 23
  12. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 23
  13. 2025Learning quadratic neural networks in high dimensions: SGD dynamics and scaling lawscited by 23
  14. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Expertscited by 21
  15. 2024Mixtures of Neural Operators Reduce Active Complexity in Operator Learningcited by 16
  16. 2020Beyond Lazy Training for Over-parameterized Tensor Decompositioncited by 15
  17. 2025Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformercited by 15
  18. 2025Mixture of Experts in Large Language Modelscited by 15
  19. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 13
  20. 2025The power of fine-grained experts: Granularity boosts expressivity in Mixture of Expertscited by 9
  21. 2024Learning Gaussian Multi-Index Models with Gradient Flow: Time Complexity and Directional Convergencecited by 8
  22. 2025When Should We Orchestrate Multiple Agents?cited by 7
  23. 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Taskscited by 4
  24. 2025Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learningcited by 4
  25. 2025Convergence Rates for Softmax Gating Mixture of Expertscited by 3
  26. 2024Learning Mixtures of Experts with EM: A Mirror Descent Perspectivecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Robustness of Mixtures of Experts to Feature Noisecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards a Statistical Understanding of Mixture-of-Experts
  2. 2026Teacher-Guided Routing for Sparse Vision Mixture-of-Experts
  3. 2026L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts
  4. 2018Deep Mixture of Experts via Shallow Embedding
  5. 2024Routers in Vision Mixture of Experts: An Empirical Study
  6. 2026MoE Lens -- An Expert Is All You Need
  7. 2026Stable Routing for Mixture-of-Experts in Class-Incremental Learning
  8. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning