# Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts

arXiv: https://arxiv.org/abs/2305.07572 · HTML: https://mlsanity.com/paper/2305.07572/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts.
- Builds on 22 of its 85 references that are in the map.
- First posted on arXiv 12 May 2023.

## Placed in

- Mixture-of-experts / [MoE theory and gating functions](https://mlsanity.com/moe/moe-theory-and-gating-functions/)

## Builds on (most cited first, 22 of 22)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts](https://mlsanity.com/paper/2206.02770/) (2022)
- [A Review of Sparse Expert Models in Deep Learning](https://mlsanity.com/paper/2209.01667/) (2022)
- [Deep Learning of Representations: Looking Forward](https://mlsanity.com/paper/1305.0445/) (2013)
- [Scalable Transfer Learning with Expert Models](https://mlsanity.com/paper/2009.13239/) (2020)
- [Convergence Rates for Gaussian Mixtures of Experts](https://mlsanity.com/paper/1907.04377/) (2019)
- [Demystifying Softmax Gating Function in Gaussian Mixture of Experts](https://mlsanity.com/paper/2305.03288/) (2023)
- [EM Converges for a Mixture of Many Linear Regressions](https://mlsanity.com/paper/1905.12106/) (2019)
- [SpeechMoE2: Mixture-of-Experts Model with Improved Routing](https://mlsanity.com/paper/2111.11831/) (2021)
- [Global Convergence of EM Algorithm for Mixtures of Two Component Linear Regression](https://mlsanity.com/paper/1810.05752/) (2018)
- [On the Minimax Optimality of the EM Algorithm for Learning Two-Component Mixed Linear Regression](https://mlsanity.com/paper/2006.02601/) (2020)
- [Refined Convergence Rates for Maximum Likelihood Estimation under Finite Mixture Models](https://mlsanity.com/paper/2202.08786/) (2022)
- [A Universal Approximation Theorem for Mixture of Experts Models](https://mlsanity.com/paper/1602.03683/) (2016)
- [High-Dimensional Regression with Gaussian Mixtures and Partially-Latent Response Variables](https://mlsanity.com/paper/1308.2302/) (2013)
- [Singularity structures and impacts on parameter estimation in finite mixtures of distributions](https://mlsanity.com/paper/1609.02655/) (2016)
- [Estimating the Number of Components in Finite Mixture Models via the Group-Sort-Fuse Procedure](https://mlsanity.com/paper/2005.11641/) (2020)
- [A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts models](https://mlsanity.com/paper/2104.02640/) (2021)
- [Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of experts](https://mlsanity.com/paper/2009.10622/) (2020)
- [Strong identifiability and parameter learning in regression with heterogeneous response](https://mlsanity.com/paper/2212.04091/) (2022)
- [Regularized Maximum Likelihood Estimation and Feature Selection in Mixtures-of-Experts Models](https://mlsanity.com/paper/1810.12161/) (2018)
- [Robust mixture of experts modeling using the t distribution](https://mlsanity.com/paper/1701.07429/) (2016)

## Built on by (most cited first, 17 of 17)

- [FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion](https://mlsanity.com/paper/2402.03226/) (2024)
- [Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts](https://mlsanity.com/paper/2309.13850/) (2023)
- [A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts](https://mlsanity.com/paper/2310.14188/) (2023)
- [CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition](https://mlsanity.com/paper/2402.02526/) (2024)
- [On Least Square Estimation in Softmax Gating Mixture of Experts](https://mlsanity.com/paper/2402.02952/) (2024)
- [Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?](https://mlsanity.com/paper/2401.13875/) (2024)
- [Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2507.11181/) (2025)
- [Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study](https://mlsanity.com/paper/2403.17404/) (2024)
- [Sparsity is Combinatorial Depth: Quantifying MoE Expressivity via Tropical Geometry](https://mlsanity.com/paper/2602.03204/) (2026)
- [On Parameter Estimation in Deviated Gaussian Mixture of Experts](https://mlsanity.com/paper/2402.05220/) (2024)
- [Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsity](https://mlsanity.com/paper/2410.13964/) (2024)
- [Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps](https://mlsanity.com/paper/2510.12744/) (2025)
- [Robustness of Mixtures of Experts to Feature Noise](https://mlsanity.com/paper/2601.14792/) (2026)
- [A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts](https://mlsanity.com/paper/2602.01468/) (2026)
- [Mixture of Many Zero-Compute Experts: A High-Rate Quantization Theory Perspective](https://mlsanity.com/paper/2510.03151/) (2025)
- [Revisiting Incremental Stochastic Majorization-Minimization Algorithms with Applications to Mixture of Experts](https://mlsanity.com/paper/2601.19811/) (2026)
- [Fast Model Selection and Stable Optimization for Softmax-Gated Multinomial-Logistic Mixture of Experts Models](https://mlsanity.com/paper/2602.07997/) (2026)

## Abstract (arXiv)

Originally introduced as a neural network for ensemble learning, mixture of experts (MoE) has recently become a fundamental building block of highly successful modern deep neural networks for heterogeneous data analysis in several applications of machine learning and statistics. Despite its popularity in practice, a satisfactory level of theoretical understanding of the MoE model is far from complete. To shed new light on this problem, we provide a convergence analysis for maximum likelihood estimation (MLE) in the Gaussian-gated MoE model. The main challenge of that analysis comes from the inclusion of covariates in the Gaussian gating functions and expert networks, which leads to their intrinsic interaction via some partial differential equations with respect to their parameters. We tackle these issues by designing novel Voronoi loss functions among parameters to accurately capture the heterogeneity of parameter estimation rates. Our findings reveal that the MLE has distinct behaviors under two complement settings of location parameters of the Gaussian gating functions, namely when all these parameters are non-zero versus when at least one among them vanishes. Notably, these behaviors can be characterized by the solvability of two different systems of polynomial equations. Finally, we conduct a simulation study to empirically verify our theoretical results.