mlsanityData to 14 Oct 2026

Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality

arXiv 2604.14419 ↗First posted 15 Apr 2026cs.AIcited by 1

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Sparse Mixture-of-Experts (MoE) architectures employ increasingly sophisticated routing mechanisms -- learned routers, multi-hop trajectories, token-dependent gating. We ask: does routing topology actually determine language modeling quality? We build a geometric MoE (ST-MoE) using cosine-similarity routing against learned centroids in a low-dimensional space (dspace = 64), requiring 80% fewer routing parameters than standard linear routers. Through 62 controlled experiments on WikiText-103 at 76--84M parameters trained to convergence (50K steps, 1.64B tokens), we find that routing topology does not determine asymptotic perplexity (PPL): five cosine-routing variants are statistically equivalent within a 1-PPL margin (Two One-Sided Tests [TOST], p < 0.05 for all 10 pairwise comparisons; 15 runs across 3 seeds, observed range 33.93--34.72). The finding extends to hash, random-fixed, and top-1 routing (single-seed; graceful 1.1--2.2 PPL degradation) and replicates on OpenWebText (0.03 PPL gap, 6 runs, 3 seeds each). A standard linear router with 5.3× more routing parameters reaches PPL 32.76, but iso-parameter cosine routing closes 67% of this gap -- the true mechanism advantage is ~1.2%. The mechanistic explanation is convergent redundancy: multi-hop updates are collinear (cos(Delta h0, Delta h1) = 0.805), implementing magnitude amplification rather than compositional reasoning; a single learnable scalar replicates multi-hop performance. As a practical payoff, zero-shot relative-norm halting saves 25% of MoE FLOPs at +0.12% PPL. Expert-level specialization and causal controllability -- which coexist with topology-level equifinality -- are explored in a companion paper.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 32 of its 38 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

32 of 38 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026MoE Lens -- An Expert Is All You Need6 Mar
  2. 2026Grouter: Decoupling Routing from Representation for Accelerated MoE Training22 Feb
  3. 2026The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models6 Jan
  4. 2025Routing Manifold Alignment Improves Generalization of Mixture-of-Experts LLMs10 Nov
  5. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing19 Dec
  6. 2024Layerwise Recurrent Router for Mixture-of-Experts13 Aug
  7. 2024Mixture of A Million Experts4 Jul
  8. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Experts23 May
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 23

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2024Mixtral of Expertscited by 1,823
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2016Deep Networks with Stochastic Depthcited by 979
  9. 2018Universal Transformerscited by 647
  10. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  11. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  12. 2016Using the Output Embedding to Improve Language Modelscited by 272
11 more
  1. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  2. 2021Hash Layers For Large Sparse Modelscited by 237
  3. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
  4. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 132
  5. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Modelscited by 45
  6. 2025Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidancecited by 26
  7. 2025Towards a Comprehensive Scaling Law of Mixture-of-Expertscited by 10
  8. 2025On Linear Mode Connectivity of Mixture-of-Experts Architecturescited by 8
  9. 2025BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inferencecited by 5
  10. 2026Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beatcited by 2
  11. 2026DirMoE: Dirichlet-routed Mixture of Expertscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
  2. 2026Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing
  3. 2026When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
  4. 2026Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
  5. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models
  6. 2026Path-Constrained Mixture-of-Experts
  7. 2026Mixture of Experts for Low-Resource LLMs
  8. 2025Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing