# Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps

arXiv: https://arxiv.org/abs/2510.12744 · HTML: https://mlsanity.com/paper/2510.12744/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts.
- Builds on 32 of its 99 references that are in the map.
- First posted on arXiv 14 Oct 2025.

## Placed in

- Mixture-of-experts / [MoE theory and gating functions](https://mlsanity.com/moe/moe-theory-and-gating-functions/)

## Builds on (most cited first, 30 of 32)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts](https://mlsanity.com/paper/2111.02358/) (2021)
- [Learning Factored Representations in a Deep Mixture of Experts](https://mlsanity.com/paper/1312.4314/) (2013)
- [Statistical guarantees for the EM algorithm: From population to sample-based analysis](https://mlsanity.com/paper/1408.2156/) (2014)
- [M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design](https://mlsanity.com/paper/2210.14793/) (2022)
- [SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts](https://mlsanity.com/paper/2105.03036/) (2021)
- [Convergence Rates for Gaussian Mixtures of Experts](https://mlsanity.com/paper/1907.04377/) (2019)
- [Singularity, Misspecification, and the Convergence Rate of EM](https://mlsanity.com/paper/1810.00828/) (2018)
- [Demystifying Softmax Gating Function in Gaussian Mixture of Experts](https://mlsanity.com/paper/2305.03288/) (2023)
- [Tight bounds for learning a mixture of two gaussians](https://mlsanity.com/paper/1404.4997/) (2014)
- [Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts](https://mlsanity.com/paper/2309.13850/) (2023)
- [Randomly initialized EM algorithm for two-component Gaussian mixture achieves near optimality in O(√n) iterations](https://mlsanity.com/paper/1908.10935/) (2019)
- [SpeechMoE2: Mixture-of-Experts Model with Improved Routing](https://mlsanity.com/paper/2111.11831/) (2021)
- [Sharp Analysis of Expectation-Maximization for Weakly Identifiable Models](https://mlsanity.com/paper/1902.00194/) (2019)
- [A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts](https://mlsanity.com/paper/2310.14188/) (2023)
- [CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition](https://mlsanity.com/paper/2402.02526/) (2024)
- [HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts](https://mlsanity.com/paper/2312.07035/) (2023)
- [A Universal Approximation Theorem for Mixture of Experts Models](https://mlsanity.com/paper/1602.03683/) (2016)
- [Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts](https://mlsanity.com/paper/2305.07572/) (2023)
- [Optimal estimation of high-dimensional location Gaussian mixtures](https://mlsanity.com/paper/2002.05818/) (2020)
- [A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts models](https://mlsanity.com/paper/2104.02640/) (2021)
- [Time series modeling by a regression approach based on a latent process](https://mlsanity.com/paper/1312.6969/) (2013)
- [Uniform Convergence Rates for Maximum Likelihood Estimation under Two-Component Gaussian Mixture Models](https://mlsanity.com/paper/2006.00704/) (2020)
- [Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of experts](https://mlsanity.com/paper/2009.10622/) (2020)
- [Strong identifiability and parameter learning in regression with heterogeneous response](https://mlsanity.com/paper/2212.04091/) (2022)
- [Non-asymptotic model selection in block-diagonal mixture of polynomial experts models](https://mlsanity.com/paper/2104.08959/) (2021)
- [Risk Bounds for Mixture Density Estimation on Compact Domains via the h-Lifted Kullback--Leibler Divergence](https://mlsanity.com/paper/2404.12586/) (2024)
- [Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures](https://mlsanity.com/paper/2505.13052/) (2025)

## Built on by (most cited first, 2 of 2)

- [Revisiting Incremental Stochastic Majorization-Minimization Algorithms with Applications to Mixture of Experts](https://mlsanity.com/paper/2601.19811/) (2026)
- [Fast Model Selection and Stable Optimization for Softmax-Gated Multinomial-Logistic Mixture of Experts Models](https://mlsanity.com/paper/2602.07997/) (2026)

## Abstract (arXiv)

We develop a unified statistical framework for softmax-gated Gaussian mixture of experts (SGMoE) that addresses three long-standing obstacles in parameter estimation and model selection: (i) non-identifiability of gating parameters up to common translations, (ii) intrinsic gate-expert interactions that induce coupled differential relations in the likelihood, and (iii) the tight numerator-denominator coupling in the softmax-induced conditional density. Our approach introduces Voronoi-type loss functions aligned with the gate-partition geometry and establishes finite-sample convergence rates for the maximum likelihood estimator (MLE). In over-specified models, we reveal a link between the MLE's convergence rate and the solvability of an associated system of polynomial equations characterizing near-nonidentifiable directions. For model selection, we adapt dendrograms of mixing measures to SGMoE, yielding a consistent, sweep-free selector of the number of experts that attains pointwise-optimal parameter rates under overfitting while avoiding multi-size training. Simulations on synthetic data corroborate the theory, accurately recovering the expert count and achieving the predicted rates for parameter estimation while closely approximating the regression function. Under model misspecification (e.g., ε-contamination), the dendrogram selection criterion is robust, recovering the true number of mixture components, while the Akaike information criterion, the Bayesian information criterion, and the integrated completed likelihood tend to overselect as sample size grows. On a maize proteomics dataset of drought-responsive traits, our dendrogram-guided SGMoE selects two experts, exposes a clear mixing-measure hierarchy, stabilizes the likelihood early, and yields interpretable genotype-phenotype maps, outperforming standard criteria without multi-size training.