# Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Experts

arXiv: https://arxiv.org/abs/2510.07205 · HTML: https://mlsanity.com/paper/2510.07205/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 39 of its 44 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 30 of 39)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [Automated Design of Agentic Systems](https://mlsanity.com/paper/2408.08435/) (2024)
- [Neural Networks can Learn Representations with Gradient Descent](https://mlsanity.com/paper/2206.15144/) (2022)
- [Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models](https://mlsanity.com/paper/2402.14800/) (2024)
- [G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks](https://mlsanity.com/paper/2410.11782/) (2024)
- [Learning Single-Index Models with Shallow Neural Networks](https://mlsanity.com/paper/2210.15651/) (2022)
- [Learning One-hidden-layer Neural Networks with Landscape Design](https://mlsanity.com/paper/1711.00501/) (2017)
- [Neural Networks Efficiently Learn Low-Dimensional Representations with SGD](https://mlsanity.com/paper/2209.14863/) (2022)
- [On Learning Gaussian Multi-index Models with Gradient Flow](https://mlsanity.com/paper/2310.19793/) (2023)
- [Theory on Mixture-of-Experts in Continual Learning](https://mlsanity.com/paper/2406.16437/) (2024)
- [Towards Understanding Mixture of Experts in Deep Learning](https://mlsanity.com/paper/2208.02813/) (2022)
- [A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Features](https://mlsanity.com/paper/2206.01717/) (2022)
- [Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit](https://mlsanity.com/paper/2406.01581/) (2024)
- [Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts](https://mlsanity.com/paper/2407.09590/) (2024)
- [Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networks](https://mlsanity.com/paper/2306.04073/) (2023)
- [Gradient-Based Feature Learning under Structured Data](https://mlsanity.com/paper/2309.03843/) (2023)
- [Emergence and scaling laws in SGD learning of shallow neural networks](https://mlsanity.com/paper/2504.19983/) (2025)
- [Learning quadratic neural networks in high dimensions: SGD dynamics and scaling laws](https://mlsanity.com/paper/2508.03688/) (2025)
- [Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts](https://mlsanity.com/paper/2405.13997/) (2024)
- [Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMs](https://mlsanity.com/paper/2502.07942/) (2025)
- [A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts](https://mlsanity.com/paper/2310.14188/) (2023)
- [On Least Square Estimation in Softmax Gating Mixture of Experts](https://mlsanity.com/paper/2402.02952/) (2024)
- [A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts](https://mlsanity.com/paper/2405.16646/) (2024)
- [Mixtures of Neural Operators Reduce Active Complexity in Operator Learning](https://mlsanity.com/paper/2404.09101/) (2024)
- [Beyond Lazy Training for Over-parameterized Tensor Decomposition](https://mlsanity.com/paper/2010.11356/) (2020)
- [Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer](https://mlsanity.com/paper/2505.24378/) (2025)

## Built on by (most cited first, 2 of 2)

- [Robustness of Mixtures of Experts to Feature Noise](https://mlsanity.com/paper/2601.14792/) (2026)
- [Dense Mixture-of-Experts as a Reparameterized Wide FFN: A Granularity Sweep at Fixed Compute](https://mlsanity.com/paper/2610.02584/) (2026)

## Abstract (arXiv)

Mixture-of-Experts (MoE) architectures have emerged as a cornerstone of modern AI systems. In particular, MoEs route inputs dynamically to specialized experts whose outputs are aggregated through weighted summation. Despite their widespread application, theoretical understanding of MoE training dynamics remains limited to either separate expert-router optimization or only top-1 routing scenarios with carefully constructed datasets. This paper advances MoE theory by providing convergence guarantees for joint training of soft-routed MoE models with non-linear routers and experts in a student-teacher framework. We prove that, with moderate over-parameterization, the student network undergoes a feature learning phase, where the router's learning process is ``guided'' by the experts, that recovers the teacher's parameters. Moreover, we show that a post-training pruning can effectively eliminate redundant neurons, followed by a provably convergent fine-tuning process that reaches global optimality. To our knowledge, our analysis is the first to bring novel insights in understanding the optimization landscape of the MoE architecture.