mlsanityData to 7 Oct 2026

Horseshoe Mixtures-of-Experts (HS-MoE)

arXiv 2601.09043 ↗First posted 14 Jan 2026stat.ML cs.LGcited by 0

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Horseshoe mixtures-of-experts (HS-MoE) models provide a Bayesian framework for sparse expert selection in mixture-of-experts architectures. We combine the horseshoe prior's adaptive global-local shrinkage with input-dependent gating, yielding data-adaptive sparsity in expert usage. Our primary methodological contribution is a particle learning algorithm for sequential inference, in which the filter is propagated forward in time while tracking only sufficient statistics. We also discuss how HS-MoE relates to modern mixture-of-experts layers in large language models, which are deployed under extreme sparsity constraints (e.g., activating a small number of experts per token out of a large pool).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 7 of its 23 references that are in the map. First posted on arXiv 14 Jan 2026.

Builds on

7 of 23 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study26 Mar

Other references · 6

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2024Mixtral of Expertscited by 1,785
  4. 2017Model Selection in Bayesian Neural Networks via Horseshoe Priorscited by 52
  5. 2015Dependent Multinomial Models Made Easy: Stick Breaking with the P'olya-Gamma Augmentationcited by 25
  6. 2025Kolmogorov GAM Networks are all you need!cited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts
  2. 2023Adaptive Gating in Mixture-of-Experts based Language Models
  3. 2026Variational Inference, Entropy, and Orthogonality: A Unified Theory of Mixture-of-Experts
  4. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  5. 2025Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks
  6. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  7. 2025Mixture of Experts in Large Language Models
  8. 2025Advancing Expert Specialization for Better MoE