mlsanityData to 7 Oct 2026

Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Function

arXiv 2602.01466 ↗First posted 1 Feb 2026stat.ML cs.LGcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

The sigmoid gate in mixture-of-experts (MoE) models has been empirically shown to outperform the softmax gate across several tasks, ranging from approximating feed-forward networks to language modeling. Additionally, recent efforts have demonstrated that the sigmoid gate is provably more sample-efficient than its softmax counterpart under regression settings. Nevertheless, there are three notable concerns that have not been addressed in the literature, namely (i) the benefits of the sigmoid gate have not been established under classification settings; (ii) existing sigmoid-gated MoE models may not converge to their ground-truth; and (iii) the effects of a temperature parameter in the sigmoid gate remain theoretically underexplored. To tackle these open problems, we perform a comprehensive analysis of multinomial logistic MoE equipped with a modified sigmoid gate to ensure model convergence. Our results indicate that the sigmoid gate exhibits a lower sample complexity than the softmax gate for both parameter and expert estimation. Furthermore, we find that incorporating a temperature into the sigmoid gate leads to a sample complexity of exponential order due to an intrinsic interaction between the temperature and gating parameters. To overcome this issue, we propose replacing the vanilla inner product score in the gating function with a Euclidean score that effectively removes that interaction, thereby substantially improving the sample complexity to a polynomial order.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 30 of its 39 references that are in the map. First posted on arXiv 1 Feb 2026.

Builds on

30 of 39 references.

References · 30

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  6. 2024Mixtral of Expertscited by 1,785
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  9. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  10. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  11. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  12. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
18 more
  1. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  2. 2024Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Expertscited by 69
  3. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
  4. 2019EM Converges for a Mixture of Many Linear Regressionscited by 27
  5. 2023Approximating Two-Layer Feedforward Networks for Efficient Transformerscited by 27
  6. 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Expertscited by 23
  7. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 23
  8. 2022Refined Convergence Rates for Maximum Likelihood Estimation under Finite Mixture Modelscited by 22
  9. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Expertscited by 16
  10. 2024Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Promptscited by 15
  11. 2025The power of fine-grained experts: Granularity boosts expressivity in Mixture of Expertscited by 9
  12. 2013Time series modeling by a regression approach based on a latent processcited by 6
  13. 2025RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Expertscited by 6
  14. 2025On DeepSeekMoE: Statistical Benefits of Shared Experts and Normalized Sigmoid Gatingcited by 5
  15. 2025On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimationcited by 4
  16. 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Taskscited by 4
  17. 2022Functional mixture-of-experts for classificationcited by 3
  18. 2024The EM-algorithm and the Method of Moments in Softmax Mixture Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Fast Model Selection and Stable Optimization for Softmax-Gated Multinomial-Logistic Mixture of Experts Models
  2. 2025Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps
  3. 2024Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?
  4. 2023Revisiting Single-gated Mixtures of Experts
  5. 2023Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts
  6. 2026On Bayesian Softmax-Gated Mixture-of-Experts Models
  7. 2024Quadratic Gating Mixture of Experts: Statistical Insights into Self-Attention
  8. 2025Convergence Rates for Softmax Gating Mixture of Experts