# Towards a Statistical Understanding of Mixture-of-Experts

arXiv: https://arxiv.org/abs/2609.03501 · HTML: https://mlsanity.com/paper/2609.03501/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 10 of its 34 references that are in the map.
- First posted on arXiv 3 Sep 2026.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 10 of 10)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [Kimi K3: Open Frontier Intelligence](https://mlsanity.com/paper/2607.24653/) (2026)
- [Mixture of A Million Experts](https://mlsanity.com/paper/2407.04153/) (2024)
- [Toward Inference-optimal Mixture-of-Expert Large Language Models](https://mlsanity.com/paper/2404.02852/) (2024)
- [When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2605.07260/) (2026)
- [Route Experts by Sequence, not by Token](https://mlsanity.com/paper/2511.06494/) (2025)
- [On DeepSeekMoE: Statistical Benefits of Shared Experts and Normalized Sigmoid Gating](https://mlsanity.com/paper/2505.10860/) (2025)
- [Combining pre-trained models via localized model averaging](https://mlsanity.com/paper/2605.13421/) (2026)

## Abstract (arXiv)

Mixture-of-experts (MoE) architectures increase model capacity by combining a collection of expert predictors through input-dependent routing, while often activating only a small subset of experts for each input. Despite their growing importance in modern large-scale models, the statistical roles of their design choices, especially routing, sparse activation, and shared experts, remain only partially understood, as existing theory has largely focused on parametric or correctly specified MoE models. In this paper, we view MoE as a form of localized aggregation and show how this localization reshapes the approximation-estimation-computation tradeoff. We derive oracle risk bounds for learning dense and sparse routing with evolving experts, separating approximation, expert-learning, and router-estimation errors, and characterize how sparse Top-K routing can retain the benefits of localized aggregation while controlling per-input computation. We also interpret gating through the geometry of input space, relating routing performance to regions of local expert advantage, and show how shared experts, as adopted in architectures such as DeepSeekMoE, can extract common predictive structure so that routed experts focus on residual local variation. Together, these results provide a unified statistical framework for understanding MoE through input-dependent expert aggregation, in which expert specialization and computational tradeoffs are governed by local predictive structure.