# Convergence Rates for Gaussian Mixtures of Experts

arXiv: https://arxiv.org/abs/1907.04377 · HTML: https://mlsanity.com/paper/1907.04377/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts.
- Builds on 5 of its 30 references that are in the map.
- First posted on arXiv 9 Jul 2019.

## Placed in

- Mixture-of-experts / [MoE theory and gating functions](https://mlsanity.com/moe/moe-theory-and-gating-functions/)

## Builds on (most cited first, 5 of 5)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Learning Factored Representations in a Deep Mixture of Experts](https://mlsanity.com/paper/1312.4314/) (2013)
- [Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithms](https://mlsanity.com/paper/1802.07417/) (2018)
- [Singularity structures and impacts on parameter estimation in finite mixtures of distributions](https://mlsanity.com/paper/1609.02655/) (2016)
- [Learning in Gated Neural Networks](https://mlsanity.com/paper/1906.02777/) (2019)

## Built on by (most cited first, 30 of 39)

- [FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion](https://mlsanity.com/paper/2402.03226/) (2024)
- [Mixture of Experts Meets Prompt-Based Continual Learning](https://mlsanity.com/paper/2405.14124/) (2024)
- [Demystifying Softmax Gating Function in Gaussian Mixture of Experts](https://mlsanity.com/paper/2305.03288/) (2023)
- [Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts](https://mlsanity.com/paper/2309.13850/) (2023)
- [Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts](https://mlsanity.com/paper/2405.13997/) (2024)
- [A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts](https://mlsanity.com/paper/2310.14188/) (2023)
- [CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition](https://mlsanity.com/paper/2402.02526/) (2024)
- [On Least Square Estimation in Softmax Gating Mixture of Experts](https://mlsanity.com/paper/2402.02952/) (2024)
- [Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts](https://mlsanity.com/paper/2305.07572/) (2023)
- [Statistical Advantages of Perturbing Cosine Router in Mixture of Experts](https://mlsanity.com/paper/2405.14131/) (2024)
- [Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?](https://mlsanity.com/paper/2401.13875/) (2024)
- [Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Prompts](https://mlsanity.com/paper/2410.02200/) (2024)
- [A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts models](https://mlsanity.com/paper/2104.02640/) (2021)
- [Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of experts](https://mlsanity.com/paper/2009.10622/) (2020)
- [Strong identifiability and parameter learning in regression with heterogeneous response](https://mlsanity.com/paper/2212.04091/) (2022)
- [Non-asymptotic model selection in block-diagonal mixture of polynomial experts models](https://mlsanity.com/paper/2104.08959/) (2021)
- [Uniform Consistency in Nonparametric Mixture Models](https://mlsanity.com/paper/2108.14003/) (2021)
- [Revisit Visual Prompt Tuning: The Expressiveness of Prompt Experts](https://mlsanity.com/paper/2501.18936/) (2025)
- [SQ Lower Bounds for Learning Mixtures of Linear Classifiers](https://mlsanity.com/paper/2310.11876/) (2023)
- [Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspective](https://mlsanity.com/paper/2502.00281/) (2025)
- [On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation](https://mlsanity.com/paper/2502.03029/) (2025)
- [Convergence Rates for Softmax Gating Mixture of Experts](https://mlsanity.com/paper/2503.03213/) (2025)
- [Mixed Regression via Approximate Message Passing](https://mlsanity.com/paper/2304.02229/) (2023)
- [Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Experts](https://mlsanity.com/paper/2402.03460/) (2024)
- [On Parameter Estimation in Deviated Gaussian Mixture of Experts](https://mlsanity.com/paper/2402.05220/) (2024)
- [Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps](https://mlsanity.com/paper/2510.12744/) (2025)
- [Joint Problems in Learning Multiple Dynamical Systems](https://mlsanity.com/paper/2311.02181/) (2023)
- [On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts](https://mlsanity.com/paper/2505.18455/) (2025)
- [Limitations of refinement methods for weak to strong generalization](https://mlsanity.com/paper/2508.17018/) (2025)
- [Robustness of Mixtures of Experts to Feature Noise](https://mlsanity.com/paper/2601.14792/) (2026)

## Abstract (arXiv)

We provide a theoretical treatment of over-specified Gaussian mixtures of experts with covariate-free gating networks. We establish the convergence rates of the maximum likelihood estimation (MLE) for these models. Our proof technique is based on a novel notion of algebraic independence of the expert functions. Drawing on optimal transport theory, we establish a connection between the algebraic independence and a certain class of partial differential equations (PDEs). Exploiting this connection allows us to derive convergence rates and minimax lower bounds for parameter estimation.