mlsanityData to 7 Oct 2026

Convergence Rates for Softmax Gating Mixture of Experts

arXiv 2503.03213 ↗First posted 5 Mar 2025stat.ML cs.LGcited by 3

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Mixture of experts (MoE) has recently emerged as an effective framework to advance the efficiency and scalability of machine learning models by softly dividing complex tasks among multiple specialized sub-models termed experts. Central to the success of MoE is an adaptive softmax gating mechanism which takes responsibility for determining the relevance of each expert to a given input and then dynamically assigning experts their respective weights. Despite its widespread use in practice, a comprehensive study on the effects of the softmax gating on the MoE has been lacking in the literature. To bridge this gap in this paper, we perform a convergence analysis of parameter estimation and expert estimation under the MoE equipped with the standard softmax gating or its variants, including a dense-to-sparse gating and a hierarchical softmax gating, respectively. Furthermore, our theories also provide useful insights into the design of sample-efficient expert structures. In particular, we demonstrate that it requires polynomially many data points to estimate experts satisfying our proposed strong identifiability condition, namely a commonly used two-layer feed-forward network. In stark contrast, estimating linear experts, which violate the strong identifiability condition, necessitates exponentially many data points as a result of intrinsic parameter interactions expressed in the language of partial differential equations. All the theoretical results are substantiated with a rigorous guarantee.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 30 of its 50 references that are in the map. First posted on arXiv 5 Mar 2025.

Builds on

30 of 50 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions3 Oct
  2. 2024On Least Square Estimation in Softmax Gating Mixture of Experts5 Feb
  3. 2024Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?25 Jan
  4. 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts22 Oct
  5. 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts25 Sep
  6. 2023Demystifying Softmax Gating Function in Gaussian Mixture of Experts5 May
  7. 2019Convergence Rates for Gaussian Mixtures of Experts9 Jul
  8. 2018Dropout Regularization in Hierarchical Mixture of Experts25 Dec

Other references · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  7. 2024Mixtral of Expertscited by 1,785
  8. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  9. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  10. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  11. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
  12. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
10 more
  1. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  2. 2024Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Expertscited by 69
  3. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
  4. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
  5. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Expertscited by 48
  6. 2021EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gatecited by 30
  7. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorizationcited by 28
  8. 2023Approximating Two-Layer Feedforward Networks for Efficient Transformerscited by 27
  9. 2022A Mixture-of-Expert Approach to RL-based Dialogue Managementcited by 16
  10. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Expertscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026φ-Balancing for Mixture-of-Experts Trainingcited by 4
  2. 2026Sparsity is Combinatorial Depth: Quantifying MoE Expressivity via Tropical Geometrycited by 3
  3. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Expertscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026On Bayesian Softmax-Gated Mixture-of-Experts Models
  2. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts
  3. 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts
  4. 2023Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts
  5. 2025On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts
  6. 2018Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithms
  7. 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
  8. 2023Improving Expert Specialization in Mixture of Experts