mlsanityData to 7 Oct 2026

Demystifying Softmax Gating Function in Gaussian Mixture of Experts

arXiv 2305.03288 ↗First posted 5 May 2023stat.ML cs.LG math.STcited by 33

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Understanding the parameter estimation of softmax gating Gaussian mixture of experts has remained a long-standing open problem in the literature. It is mainly due to three fundamental theoretical challenges associated with the softmax gating function: (i) the identifiability only up to the translation of parameters; (ii) the intrinsic interaction via partial differential equations between the softmax gating and the expert functions in the Gaussian density; (iii) the complex dependence between the numerator and denominator of the conditional density of softmax gating Gaussian mixture of experts. We resolve these challenges by proposing novel Voronoi loss functions among parameters and establishing the convergence rates of maximum likelihood estimator (MLE) for solving parameter estimation in these models. When the true number of experts is unknown and over-specified, our findings show a connection between the convergence rate of the MLE and a solvability problem of a system of polynomial equations.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 31 of its 68 references that are in the map. First posted on arXiv 5 May 2023.

Builds on

31 of 68 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning7 Jun
  2. 2019Regularized Estimation and Feature Selection in Mixtures of Generalized Linear Experts14 Jul
  3. 2019Convergence Rates for Gaussian Mixtures of Experts9 Jul
  4. 2018Regularized Maximum Likelihood Estimation and Feature Selection in Mixtures-of-Experts Models29 Oct

Other references · 27

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2021Convex optimizationcited by 2,479
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  6. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  7. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  8. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  9. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  10. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  11. 2014Statistical guarantees for the EM algorithm: From population to sample-based analysiscited by 131
  12. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
15 more
  1. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Expertscited by 48
  2. 2018Singularity, Misspecification, and the Convergence Rate of EMcited by 38
  3. 2014Tight bounds for learning a mixture of two gaussianscited by 29
  4. 2019EM Converges for a Mixture of Many Linear Regressionscited by 27
  5. 2019Randomly initialized EM algorithm for two-component Gaussian mixture achieves near optimality in O(√n) iterationscited by 27
  6. 2021SpeechMoE2: Mixture-of-Experts Model with Improved Routingcited by 26
  7. 2018Global Convergence of EM Algorithm for Mixtures of Two Component Linear Regressioncited by 25
  8. 2020On the Minimax Optimality of the EM Algorithm for Learning Two-Component Mixed Linear Regressioncited by 24
  9. 2019Sharp Analysis of Expectation-Maximization for Weakly Identifiable Modelscited by 23
  10. 2022Refined Convergence Rates for Maximum Likelihood Estimation under Finite Mixture Modelscited by 22
  11. 2020Optimal estimation of high-dimensional location Gaussian mixturescited by 16
  12. 2013Time series modeling by a regression approach based on a latent processcited by 6
  13. 2020Uniform Convergence Rates for Maximum Likelihood Estimation under Two-Component Gaussian Mixture Modelscited by 6
  14. 2022Strong identifiability and parameter learning in regression with heterogeneous responsecited by 6
  15. 2013A hidden process regression model for functional data description. Application to curve discriminationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 13; 2025: 10; 2026: 7.

33 papers.

Sort:

By area:

  1. 2024MVMoE: Multi-Task Vehicle Routing Solver with Mixture-of-Expertscited by 69
  2. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
  3. 2024Mixture of Experts Meets Prompt-Based Continual Learningcited by 43
  4. 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Expertscited by 28
  5. 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Expertscited by 23
  6. 2024On Least Square Estimation in Softmax Gating Mixture of Expertscited by 23
  7. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 23
  8. 2023Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Expertscited by 17
  9. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Expertscited by 16
  10. 2024Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?cited by 15
  11. 2024Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Promptscited by 15
  12. 2025RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Expertscited by 6
21 more
  1. 2025Revisit Visual Prompt Tuning: The Expressiveness of Prompt Expertscited by 5
  2. 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Studycited by 4
  3. 2025On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimationcited by 4
  4. 2024Fast, accurate and lightweight sequential simulation-based inference using Gaussian locally linear mappingscited by 3
  5. 2025Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspectivecited by 3
  6. 2025Convergence Rates for Softmax Gating Mixture of Expertscited by 3
  7. 2024On Parameter Estimation in Deviated Gaussian Mixture of Expertscited by 2
  8. 2024Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsitycited by 2
  9. 2025DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networkscited by 2
  10. 2025Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweepscited by 2
  11. 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Expertscited by 1
  12. 2025On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Expertscited by 1
  13. 2025Hypernetwork-Driven Low-Rank Adaptation Across Attention Headscited by 1
  14. 2026Robustness of Mixtures of Experts to Feature Noisecited by 1
  15. 2026A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Expertscited by 1
  16. 2026Soft-to-Hard Routing in Sparse Mixture-of-Experts Modelscited by 1
  17. 2025Mixture of Many Zero-Compute Experts: A High-Rate Quantization Theory Perspectivecited by 0
  18. 2026Improving Minimax Estimation Rates for Contaminated Mixture of Multinomial Logistic Experts via Expert Heterogeneitycited by 0
  19. 2026Mixture-of-Experts in Remote Sensing: A Surveycited by 0
  20. 2026FedSPM: Routing-Enabled Federated Learning under Dual Heterogeneity via Semiparametric Mixturecited by 0
  21. 2026Infinite-Parameter LLMs: Generating and Adapting Weights from Live Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026On Bayesian Softmax-Gated Mixture-of-Experts Models
  2. 2020Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of experts
  3. 2024On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions
  4. 2019Regularized Estimation and Feature Selection in Mixtures of Gaussian-Gated Experts Models
  5. 2024Quadratic Gating Mixture of Experts: Statistical Insights into Self-Attention
  6. 2026Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Function
  7. 2026Fast Model Selection and Stable Optimization for Softmax-Gated Multinomial-Logistic Mixture of Experts Models
  8. 2025Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures