mlsanityData to 14 Oct 2026

A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts

arXiv 2310.14188 ↗First posted 22 Oct 2023stat.ML cs.LGcited by 23

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Mixture-of-experts (MoE) model incorporates the power of multiple submodels via gating functions to achieve greater performance in numerous regression and classification applications. From a theoretical perspective, while there have been previous attempts to comprehend the behavior of that model under the regression settings through the convergence analysis of maximum likelihood estimation in the Gaussian MoE model, such analysis under the setting of a classification problem has remained missing in the literature. We close this gap by establishing the convergence rates of density estimation and parameter estimation in the softmax gating multinomial logistic MoE model. Notably, when part of the expert parameters vanish, these rates are shown to be slower than polynomial rates owing to an inherent interaction between the softmax gating and expert functions via partial differential equations. To address this issue, we propose using a novel class of modified softmax gating functions which transform the input before delivering them to the gating functions. As a result, the previous interaction disappears and the parameter estimation rates are significantly improved.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 28 of its 41 references that are in the map. First posted on arXiv 22 Oct 2023.

Builds on

28 of 41 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts25 Sep
  2. 2023Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts12 May
  3. 2023Demystifying Softmax Gating Function in Gaussian Mixture of Experts5 May
  4. 2022Functional mixture-of-experts for classification28 Feb
  5. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning7 Jun
  6. 2019Regularized Estimation and Feature Selection in Mixtures of Generalized Linear Experts14 Jul
  7. 2019Convergence Rates for Gaussian Mixtures of Experts9 Jul

Other references · 21

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  6. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  7. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  8. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  9. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  10. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 98
  11. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
  12. 2023Brainformers: Trading Simplicity for Efficiencycited by 31
9 more
  1. 2021EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gatecited by 30
  2. 2019EM Converges for a Mixture of Many Linear Regressionscited by 27
  3. 2021Probabilistic Mixture-of-Experts for Efficient Deep Reinforcement Learningcited by 27
  4. 2021SpeechMoE2: Mixture-of-Experts Model with Improved Routingcited by 26
  5. 2018Global Convergence of EM Algorithm for Mixtures of Two Component Linear Regressioncited by 25
  6. 2020On the Minimax Optimality of the EM Algorithm for Learning Two-Component Mixed Linear Regressioncited by 24
  7. 2022Refined Convergence Rates for Maximum Likelihood Estimation under Finite Mixture Modelscited by 22
  8. 2022A Mixture-of-Expert Approach to RL-based Dialogue Managementcited by 16
  9. 2013High-Dimensional Regression with Gaussian Mixtures and Partially-Latent Response Variablescited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 7; 2026: 10.

23 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  2. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 68
  3. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 24
  4. 2024On Least Square Estimation in Softmax Gating Mixture of Expertscited by 23
  5. 2024Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?cited by 15
  6. 2025RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Expertscited by 6
  7. 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Studycited by 5
  8. 2025On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimationcited by 4
  9. 2025Convergence Rates for Softmax Gating Mixture of Expertscited by 4
  10. 2026Sparsity is Combinatorial Depth: Quantifying MoE Expressivity via Tropical Geometrycited by 3
  11. 2024Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsitycited by 2
  12. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Expertscited by 2
11 more
  1. 2025Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweepscited by 2
  2. 2025On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Expertscited by 1
  3. 2026Robustness of Mixtures of Experts to Feature Noisecited by 1
  4. 2026A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Expertscited by 1
  5. 2026Soft-to-Hard Routing in Sparse Mixture-of-Experts Modelscited by 1
  6. 2026Revisiting Incremental Stochastic Majorization-Minimization Algorithms with Applications to Mixture of Expertscited by 0
  7. 2026Improving Minimax Estimation Rates for Contaminated Mixture of Multinomial Logistic Experts via Expert Heterogeneitycited by 0
  8. 2026Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Functioncited by 0
  9. 2026Fast Model Selection and Stable Optimization for Softmax-Gated Multinomial-Logistic Mixture of Experts Modelscited by 0
  10. 2026Mixture-of-Experts in Remote Sensing: A Surveycited by 0
  11. 2026Structure Aware Neural Architecture Search for Mixture of Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026On Bayesian Softmax-Gated Mixture-of-Experts Models
  2. 2020Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of experts
  3. 2024On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions
  4. 2019Regularized Estimation and Feature Selection in Mixtures of Gaussian-Gated Experts Models
  5. 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts
  6. 2018Regularized Maximum Likelihood Estimation and Feature Selection in Mixtures-of-Experts Models
  7. 2024Quadratic Gating Mixture of Experts: Statistical Insights into Self-Attention
  8. 2025Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures