Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps
On the map: Mixture-of-experts → MoE theory and gating functions
Abstract (arXiv)
We develop a unified statistical framework for softmax-gated Gaussian mixture of experts (SGMoE) that addresses three long-standing obstacles in parameter estimation and model selection: (i) non-identifiability of gating parameters up to common translations, (ii) intrinsic gate-expert interactions that induce coupled differential relations in the likelihood, and (iii) the tight numerator-denominator coupling in the softmax-induced conditional density. Our approach introduces Voronoi-type loss functions aligned with the gate-partition geometry and establishes finite-sample convergence rates for the maximum likelihood estimator (MLE). In over-specified models, we reveal a link between the MLE's convergence rate and the solvability of an associated system of polynomial equations characterizing near-nonidentifiable directions. For model selection, we adapt dendrograms of mixing measures to SGMoE, yielding a consistent, sweep-free selector of the number of experts that attains pointwise-optimal parameter rates under overfitting while avoiding multi-size training. Simulations on synthetic data corroborate the theory, accurately recovering the expert count and achieving the predicted rates for parameter estimation while closely approximating the regression function. Under model misspecification (e.g., ε-contamination), the dendrogram selection criterion is robust, recovering the true number of mixture components, while the Akaike information criterion, the Bayesian information criterion, and the integrated completed likelihood tend to overselect as sample size grows. On a maize proteomics dataset of drought-responsive traits, our dendrogram-guided SGMoE selects two experts, exposes a clear mixing-measure hierarchy, stabilizes the likelihood early, and yields interpretable genotype-phenotype maps, outperforming standard criteria without multi-size training.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 32 of its 99 references that are in the map. First posted on arXiv 14 Oct 2025.
Builds on
32 of 99 references.
Closest ancestors · 8
Same direction, newest first
- 2025Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures19 May
- 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts22 Oct
- 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts25 Sep
- 2023Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts12 May
- 2023Demystifying Softmax Gating Function in Gaussian Mixture of Experts5 May
- 2021A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts models6 Apr
- 2020Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of experts22 Sep
- 2019Convergence Rates for Gaussian Mixtures of Experts9 Jul
Other references · 24
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
- 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
- 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
- 2014Statistical guarantees for the EM algorithm: From population to sample-based analysiscited by 133
- 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 98
- 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Expertscited by 48
- 2018Singularity, Misspecification, and the Convergence Rate of EMcited by 38
- 2014Tight bounds for learning a mixture of two gaussianscited by 29
- 2019Randomly initialized EM algorithm for two-component Gaussian mixture achieves near optimality in O(√n) iterationscited by 27
- 2021SpeechMoE2: Mixture-of-Experts Model with Improved Routingcited by 26
12 more
- 2019Sharp Analysis of Expectation-Maximization for Weakly Identifiable Modelscited by 23
- 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competitioncited by 23
- 2023HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Expertscited by 21
- 2016A Universal Approximation Theorem for Mixture of Experts Modelscited by 18
- 2020Optimal estimation of high-dimensional location Gaussian mixturescited by 16
- 2013Time series modeling by a regression approach based on a latent processcited by 6
- 2020Uniform Convergence Rates for Maximum Likelihood Estimation under Two-Component Gaussian Mixture Modelscited by 6
- 2022Strong identifiability and parameter learning in regression with heterogeneous responsecited by 6
- 2021Non-asymptotic model selection in block-diagonal mixture of polynomial experts modelscited by 5
- 2024Risk Bounds for Mixture Density Estimation on Compact Domains via the h-Lifted Kullback--Leibler Divergencecited by 3
- 2021Identifying Brexit voting patterns in the British House of Commons: an analysis based on Bayesian mixture models with flexible concomitant covariate effectscited by 2
- 2025A Unified Framework for Variable Selection in Model-Based Clustering with Missing Not at Randomcited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026On Bayesian Softmax-Gated Mixture-of-Experts Models
- 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts
- 2026Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Function
- 2019Regularized Estimation and Feature Selection in Mixtures of Generalized Linear Experts
- 2024Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?
- 2025Convergence Rates for Softmax Gating Mixture of Experts
- 2024On Least Square Estimation in Softmax Gating Mixture of Experts
- 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts