mlsanityData to 7 Oct 2026

On Least Square Estimation in Softmax Gating Mixture of Experts

arXiv 2402.02952 ↗First posted 5 Feb 2024stat.ML cs.LGcited by 23

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Mixture of experts (MoE) model is a statistical machine learning design that aggregates multiple expert networks using a softmax gating function in order to form a more intricate and expressive model. Despite being commonly used in several applications owing to their scalability, the mathematical and statistical properties of MoE models are complex and difficult to analyze. As a result, previous theoretical works have primarily focused on probabilistic MoE models by imposing the impractical assumption that the data are generated from a Gaussian MoE model. In this work, we investigate the performance of the least squares estimators (LSE) under a deterministic MoE model where the data are sampled according to a regression model, a setting that has remained largely unexplored. We establish a condition called strong identifiability to characterize the convergence behavior of various types of expert functions. We demonstrate that the rates for estimating strongly identifiable experts, namely the widely used feed-forward networks with activation functions sigmoid(·) and tanh(·), are substantially faster than those of polynomial experts, which we show to exhibit a surprising slow estimation rate. Our findings have important practical implications for expert selection.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 23 of its 39 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

23 of 39 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts22 Oct
  2. 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts25 Sep
  3. 2023Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts12 May
  4. 2023Demystifying Softmax Gating Function in Gaussian Mixture of Experts5 May
  5. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning7 Jun
  6. 2019Convergence Rates for Gaussian Mixtures of Experts9 Jul

Other references · 17

  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2024Mixtral of Expertscited by 1,785
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  6. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  7. 2023From Sparse to Soft Mixtures of Expertscited by 245
  8. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  9. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
  10. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  11. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
  12. 2023Brainformers: Trading Simplicity for Efficiencycited by 31
5 more
  1. 2021EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gatecited by 30
  2. 2023Approximating Two-Layer Feedforward Networks for Efficient Transformerscited by 27
  3. 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competitioncited by 23
  4. 2022Refined Convergence Rates for Maximum Likelihood Estimation under Finite Mixture Modelscited by 22
  5. 2022A Mixture-of-Expert Approach to RL-based Dialogue Managementcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 13; 2026: 3.

23 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  2. 2024Mixture of Experts Meets Prompt-Based Continual Learningcited by 43
  3. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 23
  4. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Expertscited by 16
  5. 2024Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Promptscited by 15
  6. 2025Tight Clusters Make Specialized Expertscited by 8
  7. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
  8. 2025RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Expertscited by 6
  9. 2025Understanding the Mixture-of-Experts with Nadaraya-Watson Kernelcited by 6
  10. 2025Revisit Visual Prompt Tuning: The Expressiveness of Prompt Expertscited by 5
  11. 2025Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspectivecited by 3
  12. 2025Convergence Rates for Softmax Gating Mixture of Expertscited by 3
11 more
  1. 2024On Parameter Estimation in Deviated Gaussian Mixture of Expertscited by 2
  2. 2025DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networkscited by 2
  3. 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Expertscited by 1
  4. 2025STM3: Mixture of Multiscale Mamba for Long-Term Spatio-Temporal Time-Series Predictioncited by 1
  5. 2025Limitations of refinement methods for weak to strong generalizationcited by 1
  6. 2025Hypernetwork-Driven Low-Rank Adaptation Across Attention Headscited by 1
  7. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Expertscited by 1
  8. 2026Soft-to-Hard Routing in Sparse Mixture-of-Experts Modelscited by 1
  9. 2025Mixture of Many Zero-Compute Experts: A High-Rate Quantization Theory Perspectivecited by 0
  10. 2026OTSS: Output-Targeted Soft Segmentation for Contextual Decision-Weight Learningcited by 0
  11. 2026Task-Aware Federated Fine-Tuning for MoE-based Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of experts
  2. 2025On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts
  3. 2026On Bayesian Softmax-Gated Mixture-of-Experts Models
  4. 2019Regularized Estimation and Feature Selection in Mixtures of Generalized Linear Experts
  5. 2024Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?
  6. 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
  7. 2017An Introduction to the Practical and Theoretical Aspects of Mixture-of-Experts Modeling
  8. 2025Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps