# On Parameter Estimation in Deviated Gaussian Mixture of Experts

arXiv: https://arxiv.org/abs/2402.05220 · HTML: https://mlsanity.com/paper/2402.05220/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts.
- Builds on 22 of its 63 references that are in the map.
- First posted on arXiv 7 Feb 2024.

## Placed in

- Mixture-of-experts / [MoE theory and gating functions](https://mlsanity.com/moe/moe-theory-and-gating-functions/)

## Builds on (most cited first, 22 of 22)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [Learning Factored Representations in a Deep Mixture of Experts](https://mlsanity.com/paper/1312.4314/) (2013)
- [DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning](https://mlsanity.com/paper/2106.03760/) (2021)
- [M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design](https://mlsanity.com/paper/2210.14793/) (2022)
- [FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion](https://mlsanity.com/paper/2402.03226/) (2024)
- [Scalable Transfer Learning with Expert Models](https://mlsanity.com/paper/2009.13239/) (2020)
- [Convergence Rates for Gaussian Mixtures of Experts](https://mlsanity.com/paper/1907.04377/) (2019)
- [MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation](https://mlsanity.com/paper/2204.07675/) (2022)
- [Demystifying Softmax Gating Function in Gaussian Mixture of Experts](https://mlsanity.com/paper/2305.03288/) (2023)
- [Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts](https://mlsanity.com/paper/2309.13850/) (2023)
- [SpeechMoE2: Mixture-of-Experts Model with Improved Routing](https://mlsanity.com/paper/2111.11831/) (2021)
- [CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition](https://mlsanity.com/paper/2402.02526/) (2024)
- [On Least Square Estimation in Softmax Gating Mixture of Experts](https://mlsanity.com/paper/2402.02952/) (2024)
- [Testing Goodness of Fit of Conditional Density Models with Kernels](https://mlsanity.com/paper/2002.10271/) (2020)
- [Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts](https://mlsanity.com/paper/2305.07572/) (2023)
- [Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?](https://mlsanity.com/paper/2401.13875/) (2024)
- [Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithms](https://mlsanity.com/paper/1802.07417/) (2018)
- [Detection and Feature Selection in Sparse Mixture Models](https://mlsanity.com/paper/1405.1478/) (2014)
- [Learning in Gated Neural Networks](https://mlsanity.com/paper/1906.02777/) (2019)
- [Two-component Mixture Model in the Presence of Covariates](https://mlsanity.com/paper/1810.07897/) (2018)

## Built on by (most cited first, 2 of 2)

- [On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts](https://mlsanity.com/paper/2505.18455/) (2025)
- [Improving Minimax Estimation Rates for Contaminated Mixture of Multinomial Logistic Experts via Expert Heterogeneity](https://mlsanity.com/paper/2602.00939/) (2026)

## Abstract (arXiv)

We consider the parameter estimation problem in the deviated Gaussian mixture of experts in which the data are generated from (1 - λast) g0(Y| X)+ λast sumi = 1^kast piast f(Y|(aiast)topX+biast,σiast), where X, Y are respectively a covariate vector and a response variable, g0(Y|X) is a known function, λast ∈ [0, 1] is true but unknown mixing proportion, and (piast, aiast, biast, σiast) for 1 ≤ i ≤ kast are unknown parameters of the Gaussian mixture of experts. This problem arises from the goodness-of-fit test when we would like to test whether the data are generated from g0(Y|X) (null hypothesis) or they are generated from the whole mixture (alternative hypothesis). Based on the algebraic structure of the expert functions and the distinguishability between g0 and the mixture part, we construct novel Voronoi-based loss functions to capture the convergence rates of maximum likelihood estimation (MLE) for our models. We further demonstrate that our proposed loss functions characterize the local convergence rates of parameter estimation more accurately than the generalized Wasserstein, a loss function being commonly used for estimating parameters in the Gaussian mixture of experts.