mlsanityData to 7 Oct 2026

On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions

arXiv 2410.02935 ↗First posted 3 Oct 2024stat.ML cs.LGcited by 16

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

With the growing prominence of the Mixture of Experts (MoE) architecture in developing large-scale foundation models, we investigate the Hierarchical Mixture of Experts (HMoE), a specialized variant of MoE that excels in handling complex inputs and improving performance on targeted tasks. Our analysis highlights the advantages of using the Laplace gating function over the traditional Softmax gating within the HMoE frameworks. We theoretically demonstrate that applying the Laplace gating function at both levels of the HMoE model helps eliminate undesirable parameter interactions caused by the Softmax gating and, therefore, accelerates the expert convergence as well as enhances the expert specialization. Empirical validation across diverse scenarios supports these theoretical claims. This includes large-scale multimodal tasks, image classification, and latent domain discovery and prediction tasks, where our modified HMoE models show great performance improvements compared to the conventional HMoE models.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 13 of its 98 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

13 of 98 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Improving Expert Specialization in Mixture of Experts28 Feb

Other references · 12

  1. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  2. 2019Invariant Risk Minimizationcited by 1,832
  3. 2024Mixtral of Expertscited by 1,785
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  5. 2019Publicly Available Clinical BERT Embeddingscited by 995
  6. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  7. 2021Multi-Time Attention Networks for Irregularly Sampled Time Seriescited by 234
  8. 2019Time2Vec: Learning a Vector Representation of Timecited by 208
  9. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  10. 2022UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognitioncited by 61
  11. 2021EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gatecited by 30
  12. 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competitioncited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 5.

16 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  2. 2025Mixture of Experts in Large Language Modelscited by 15
  3. 2025Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Modelcited by 6
  4. 2025Towards Interpretability Without Sacrifice: Faithful Dense Layer Decomposition with Mixture of Decoderscited by 5
  5. 2024Quadratic Gating Mixture of Experts: Statistical Insights into Self-Attentioncited by 4
  6. 2025On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimationcited by 4
  7. 2025Massively Multimodal Foundation Models: A Framework for Capturing Interactions with Specialized Mixture-of-Expertscited by 4
  8. 2025Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspectivecited by 3
  9. 2025Convergence Rates for Softmax Gating Mixture of Expertscited by 3
  10. 2025DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networkscited by 2
  11. 2026Hierarchical Mixture-of-Experts with Two-Stage Optimizationcited by 2
  12. 2025On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Expertscited by 1
4 more
  1. 2026A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Expertscited by 1
  2. 2026On Bayesian Softmax-Gated Mixture-of-Experts Modelscited by 1
  3. 2026FLAME: Adaptive Mixture-of-Experts for Continual Multimodal Multi-Task Learningcited by 1
  4. 2026Counterfactual Self-Evolving Agents for Evidence-Grounded Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts
  2. 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts
  3. 2026Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization
  4. 2025Rethinking Sparse Mixture of Experts from a Unified Perspective
  5. 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts
  6. 2026Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization
  7. 2026Horseshoe Mixtures-of-Experts (HS-MoE)
  8. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization