mlsanityData to 14 Oct 2026

E = T*H/(O+B): A Dimensionless Control Parameter for Mixture-of-Experts Ecology

arXiv 2605.06415 ↗First posted 7 May 2026cs.LG cs.AI cs.CLcited by 0

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

We introduce E = T*H/(O+B), a dimensionless control parameter that predicts whether Mixture-of-Experts (MoE) models will develop a healthy expert ecology or collapse into dead experts. E combines four hyperparameters -- routing temperature T, routing entropy weight H, oracle weight O, and balance weight B -- into a single quantity. Through 12 controlled experiments (8 vision, 4 language) totaling over 11,000 training epochs, we establish that E >= 0.5 alone is sufficient to guarantee zero dead experts, removing the necessity for handcrafted load-balancing auxiliary losses. We validate this cross-modally on CIFAR-10, CIFAR-100, TinyImageNet-200, WikiText-2, and WikiText-103. Six additional findings emerge: (1) dead experts can resuscitate -- triggered by balance loss driving router re-exploration; (2) ortho toxicity is dataset-dependent, not universal; (3) task complexity shifts the critical E threshold; (4) model overfitting is decoupled from expert ecological health; (5) three-tier MoE spontaneously collapses into a two-tier functional structure; (6) ecological structure is temperature-invariant across a 50x range. We propose that E serves as a unified diagnostic for MoE training, analogous to the Reynolds number in fluid dynamics.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 11 of its 20 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

11 of 20 references.

References · 11

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2016Wide Residual Networkscited by 3,070
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  5. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  6. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  7. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  8. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  9. 2023From Sparse to Soft Mixtures of Expertscited by 248
  10. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
  11. 2025Advancing Expert Specialization for Better MoEcited by 39

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms
  2. 2026Hierarchical Mixture-of-Experts with Two-Stage Optimization
  3. 2025Let the Experts Speak: Improving Survival Prediction & Calibration via Mixture-of-Experts Heads
  4. 2026From Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts Models
  5. 2025MixtureKit: A General Framework for Composing, Training, and Visualizing Mixture-of-Experts Models
  6. 2026More Experts, Worse Dynamics: Inverse Scaling and Spectral Bias in Mixture-of-Experts State-Space Models
  7. 2026Towards a Statistical Understanding of Mixture-of-Experts
  8. 2026Dense Mixture-of-Experts as a Reparameterized Wide FFN: A Granularity Sweep at Fixed Compute