mlsanityData to 7 Oct 2026

Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts

arXiv 2405.13997 ↗First posted 22 May 2024stat.ML cs.LGcited by 23

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

The softmax gating function is arguably the most popular choice in mixture of experts modeling. Despite its widespread use in practice, the softmax gating may lead to unnecessary competition among experts, potentially causing the undesirable phenomenon of representation collapse due to its inherent structure. In response, the sigmoid gating function has been recently proposed as an alternative and has been demonstrated empirically to achieve superior performance. However, a rigorous examination of the sigmoid gating function is lacking in current literature. In this paper, we verify theoretically that the sigmoid gating, in fact, enjoys a higher sample efficiency than the softmax gating for the statistical task of expert estimation. Towards that goal, we consider a regression framework in which the unknown regression function is modeled as a mixture of experts, and study the rates of convergence of the least squares estimator under the over-specified case in which the number of fitted experts is larger than the true value. We show that two gating regimes naturally arise and, in each of them, we formulate an identifiability condition for the expert functions and derive the corresponding convergence rates. In both cases, we find that experts formulated as feed-forward networks with commonly used activation such as ReLU and GELU enjoy faster convergence rates under the sigmoid gating than those under softmax gating. Furthermore, given the same choice of experts, we demonstrate that the sigmoid gating function requires a smaller sample size than its softmax counterpart to attain the same error of expert estimation and, therefore, is more sample efficient.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 29 of its 39 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

29 of 39 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024On Least Square Estimation in Softmax Gating Mixture of Experts5 Feb
  2. 2024Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?25 Jan
  3. 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts22 Oct
  4. 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts25 Sep
  5. 2023Demystifying Softmax Gating Function in Gaussian Mixture of Experts5 May
  6. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning7 Jun
  7. 2019Convergence Rates for Gaussian Mixtures of Experts9 Jul

Other references · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2024Mixtral of Expertscited by 1,785
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  8. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  9. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  10. 2023From Sparse to Soft Mixtures of Expertscited by 245
  11. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  12. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
10 more
  1. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  2. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
  3. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
  4. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Expertscited by 48
  5. 2023Brainformers: Trading Simplicity for Efficiencycited by 31
  6. 2023Approximating Two-Layer Feedforward Networks for Efficient Transformerscited by 27
  7. 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competitioncited by 23
  8. 2022Refined Convergence Rates for Maximum Likelihood Estimation under Finite Mixture Modelscited by 22
  9. 2022A Mixture-of-Expert Approach to RL-based Dialogue Managementcited by 16
  10. 2022Sparse Mixers: Combining MoE and Mixing to build a more efficient BERTcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 10; 2026: 9.

23 papers.

Sort:

By area:

  1. 2024Upcycling Large Language Models into Mixture of Expertscited by 48
  2. 2024Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Promptscited by 15
  3. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
  4. 2025RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Expertscited by 6
  5. 2025Understanding the Mixture-of-Experts with Nadaraya-Watson Kernelcited by 6
  6. 2025Revisit Visual Prompt Tuning: The Expressiveness of Prompt Expertscited by 5
  7. 2025On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimationcited by 4
  8. 2026φ-Balancing for Mixture-of-Experts Trainingcited by 4
  9. 2026Mamoda2.5: Enhancing Unified Multimodal Model with DiT-MoEcited by 3
  10. 2024Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsitycited by 2
  11. 2025DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networkscited by 2
  12. 2026JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulationcited by 2
11 more
  1. 2025On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Expertscited by 1
  2. 2025Limitations of refinement methods for weak to strong generalizationcited by 1
  3. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Expertscited by 1
  4. 2025SAGE: Shape-Adapting Gated Experts for Adaptive Histopathology Image Segmentationcited by 1
  5. 2026Soft-to-Hard Routing in Sparse Mixture-of-Experts Modelscited by 1
  6. 2026MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understandingcited by 1
  7. 2025Mixture of Many Zero-Compute Experts: A High-Rate Quantization Theory Perspectivecited by 0
  8. 2026Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Functioncited by 0
  9. 2026Suppressing Non-Semantic Noise in Masked Image Modeling Representationscited by 0
  10. 2026OTSS: Output-Targeted Soft Segmentation for Contextual Decision-Weight Learningcited by 0
  11. 2026Fidelity-Diversity Metrics for Textcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026On Bayesian Softmax-Gated Mixture-of-Experts Models
  2. 2025Convergence Rates for Softmax Gating Mixture of Experts
  3. 2025Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps
  4. 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts
  5. 2024On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions
  6. 2023Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts
  7. 2024Quadratic Gating Mixture of Experts: Statistical Insights into Self-Attention
  8. 2019Regularized Estimation and Feature Selection in Mixtures of Generalized Linear Experts