# On Bayesian Softmax-Gated Mixture-of-Experts Models

arXiv: https://arxiv.org/abs/2604.20551 · HTML: https://mlsanity.com/paper/2604.20551/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts.
- Builds on 8 of its 89 references that are in the map.
- First posted on arXiv 22 Apr 2026.

## Placed in

- Mixture-of-experts / [MoE theory and gating functions](https://mlsanity.com/moe/moe-theory-and-gating-functions/)

## Builds on (most cited first, 8 of 8)

- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning](https://mlsanity.com/paper/1704.00805/) (2017)
- [On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions](https://mlsanity.com/paper/2410.02935/) (2024)
- [TabMGP: Martingale Posterior with TabPFN](https://mlsanity.com/paper/2510.25154/) (2025)
- [Estimating the Number of Components in Finite Mixture Models via Variational Approximation](https://mlsanity.com/paper/2404.16746/) (2024)
- [Scalable Posterior Uncertainty for Flexible Density-Based Clustering](https://mlsanity.com/paper/2603.03188/) (2026)

## Built on by (most cited first, 1 of 1)

- [A Heterogeneous Mixture of Experts Framework for Interpretable Machine Learning](https://mlsanity.com/paper/2608.24195/) (2026)

## Abstract (arXiv)

Mixture-of-experts models provide a flexible framework for learning complex probabilistic input-output relationships by combining multiple expert models through an input-dependent gating mechanism. These models have become increasingly prominent in modern machine learning, yet their theoretical properties in the Bayesian framework remain largely unexplored. In this paper, we study Bayesian mixture-of-experts models, focusing on the ubiquitous softmax-based gating mechanism. Specifically, we investigate the asymptotic behavior of the posterior distribution for three fundamental statistical tasks: density estimation, parameter estimation, and model selection. First, we establish posterior contraction rates for density estimation, both in the regimes with a fixed, known number of experts and with a random learnable number of experts. We then analyze parameter estimation and derive convergence guarantees based on tailored Voronoi-type losses, which account for the complex identifiability structure of mixture-of-experts models. Finally, we propose and analyze two complementary strategies for selecting the number of experts. Taken together, these results provide one of the first systematic theoretical analyses of Bayesian mixture-of-experts models with softmax gating, and yield several theory-grounded insights for practical model design.