mlsanityData to 14 Oct 2026

A Minimal Bifurcation Model of Load Imbalance in a Softmax Mixture-of-Experts Router

arXiv 2605.29121 ↗First posted 27 May 2026math.DS cs.AI cs.LGcited by 1

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

We propose a minimal dynamical model of adaptive softmax routing for a two-expert Mixture-of-Experts (MoE) layer. The model is obtained as a mean-field limit of a discrete reinforcement rule: the selected expert receives a small score increment, while all scores undergo regularizing decay. In the symmetric case the limiting system has a supercritical pitchfork bifurcation: for weak feedback there is a unique stable balanced state, whereas above a critical feedback strength two stable asymmetric states appear. When an external asymmetry is added, the pitchfork unfolds into a pair of fold bifurcations forming a cusp in the control-parameter plane. We derive exact parametric equations for the bifurcation set and the local normal form of the cusp catastrophe. Numerical experiments connect this picture to empirical expert load, a small trainable MoE model, hard top-1 PyTorch routing, and a small classification experiment on digits. The results provide a controlled low-dimensional mechanism for abrupt transitions to load imbalance in adaptive MoE routers.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 13 of its 21 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

13 of 21 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Soft-to-Hard Routing in Sparse Mixture-of-Experts Models4 May
  2. 2026Three Phases of Expert Routing: How Load Balance Evolves During Mixture-of-Experts Training5 Apr
  3. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts28 Aug
  4. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 8

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  6. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  7. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  8. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026How Shared Representations Become Specialized: Formation, Maintenance, and Latent Susceptibilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Routing-Free Mixture-of-Experts
  2. 2026Router Prior Bias: Preserving Base Routing Structure in MoE Post-Training
  3. 2026Beyond Routing: Decoupling Expert Dispatch and Aggregation in Sparse Mixture-of-Experts
  4. 2026Hierarchical Copula-Gumbel-Top-K Routing: Two-Sided Dependence Control for Frozen Mixture-of-Experts at Fixed Per-Token Routing Laws
  5. 2026Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts
  6. 2025FURINA: Free from Unmergeable Router via LINear Aggregation of mixed experts
  7. 2026A Persistent State for Auditable Mixture-of-Experts Routing
  8. 2025Load Balancing Mixture of Experts with Similarity Preserving Routers