mlsanityData to 7 Oct 2026

DirMoE: Dirichlet-routed Mixture of Experts

arXiv 2602.09001 ↗First posted 9 Feb 2026cs.LGcited by 1

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-Experts (MoE) models have demonstrated exceptional performance in large-scale language models. Existing routers typically rely on non-differentiable Top-k+Softmax, limiting their performance and scalability. We argue that two distinct decisions, which experts to activate and how to distribute expert contributions among them, are conflated in standard Top-k+Softmax. We introduce Dirichlet-Routed MoE (DirMoE), a novel end-to-end differentiable routing mechanism built on a Dirichlet variational autoencoder framework. This design fundamentally disentangles the core routing problems: expert selection, modeled by a Bernoulli component, and expert contribution among chosen experts, handled by a Dirichlet component. The entire forward pass remains fully differentiable through the use of Gumbel-Sigmoid relaxation for the expert selection and implicit reparameterization for the Dirichlet distribution. Our training objective, a variational ELBO, includes a direct sparsity penalty that precisely controls the number of active experts in expectation, alongside a schedule for key hyperparameters that guides the model from an exploratory to a definitive routing state. Moreover, our DirMoE router matches or exceeds other methods while improving expert specialization.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 42 of its 48 references that are in the map. First posted on arXiv 9 Feb 2026.

Builds on

42 of 48 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models10 Sep
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  3. 2024Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training6 May
  4. 2024Mixtral of Experts8 Jan
  5. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  6. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 34

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
22 more
  1. 2020GLU Variants Improve Transformercited by 1,724
  2. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  3. 2019Root Mean Square Layer Normalizationcited by 1,425
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  5. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  7. 2016Deep Variational Information Bottleneckcited by 915
  8. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  9. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  10. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  11. 2023From Sparse to Soft Mixtures of Expertscited by 245
  12. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  13. 2021Hash Layers For Large Sparse Modelscited by 236
  14. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  15. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 191
  16. 2018Implicit Reparameterization Gradientscited by 84
  17. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 55
  18. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
  19. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
  20. 2023Sparse Backpropagation for MoE Trainingcited by 16
  21. 2024Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformerscited by 8
  22. 2024Attention Is All You Need For Mixture-of-Depths Routingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Qualitycited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs
  2. 2025ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts
  3. 2026Variational Inference, Entropy, and Orthogonality: A Unified Theory of Mixture-of-Experts
  4. 2025Bayesian Mixture-of-Experts: Towards Making LLMs Know What They Don't Know
  5. 2025Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEs
  6. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  7. 2026DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
  8. 2025Advancing Expert Specialization for Better MoE