Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality
On the map: Mixture-of-experts → Expert routing strategies
Abstract (arXiv)
Sparse Mixture-of-Experts (MoE) architectures employ increasingly sophisticated routing mechanisms -- learned routers, multi-hop trajectories, token-dependent gating. We ask: does routing topology actually determine language modeling quality? We build a geometric MoE (ST-MoE) using cosine-similarity routing against learned centroids in a low-dimensional space (dspace = 64), requiring 80% fewer routing parameters than standard linear routers. Through 62 controlled experiments on WikiText-103 at 76--84M parameters trained to convergence (50K steps, 1.64B tokens), we find that routing topology does not determine asymptotic perplexity (PPL): five cosine-routing variants are statistically equivalent within a 1-PPL margin (Two One-Sided Tests [TOST], p < 0.05 for all 10 pairwise comparisons; 15 runs across 3 seeds, observed range 33.93--34.72). The finding extends to hash, random-fixed, and top-1 routing (single-seed; graceful 1.1--2.2 PPL degradation) and replicates on OpenWebText (0.03 PPL gap, 6 runs, 3 seeds each). A standard linear router with 5.3× more routing parameters reaches PPL 32.76, but iso-parameter cosine routing closes 67% of this gap -- the true mechanism advantage is ~1.2%. The mechanistic explanation is convergent redundancy: multi-hop updates are collinear (cos(Delta h0, Delta h1) = 0.805), implementing magnitude amplification rather than compositional reasoning; a single learnable scalar replicates multi-hop performance. As a practical payoff, zero-shot relative-norm halting saves 25% of MoE FLOPs at +0.12% PPL. Expert-level specialization and causal controllability -- which coexist with topology-level equifinality -- are explored in a companion paper.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 32 of its 38 references that are in the map. First posted on arXiv 15 Apr 2026.
Builds on
32 of 38 references.
Closest ancestors · 9
Same direction, newest first
- 2026MoE Lens -- An Expert Is All You Need6 Mar
- 2026Grouter: Decoupling Routing from Representation for Accelerated MoE Training22 Feb
- 2026The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models6 Jan
- 2025Routing Manifold Alignment Improves Generalization of Mixture-of-Experts LLMs10 Nov
- 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing19 Dec
- 2024Layerwise Recurrent Router for Mixture-of-Experts13 Aug
- 2024Mixture of A Million Experts4 Jul
- 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Experts23 May
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan
Other references · 23
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2024Mixtral of Expertscited by 1,823
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
- 2016Deep Networks with Stochastic Depthcited by 979
- 2018Universal Transformerscited by 647
- 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
- 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
- 2016Using the Output Embedding to Improve Language Modelscited by 272
11 more
- 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
- 2021Hash Layers For Large Sparse Modelscited by 237
- 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
- 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 132
- 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Modelscited by 45
- 2025Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidancecited by 26
- 2025Towards a Comprehensive Scaling Law of Mixture-of-Expertscited by 10
- 2025On Linear Mode Connectivity of Mixture-of-Experts Architecturescited by 8
- 2025BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inferencecited by 5
- 2026Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beatcited by 2
- 2026DirMoE: Dirichlet-routed Mixture of Expertscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026Mixture of Layers with Hybrid Attentioncited by 0
- 2026Mixture of Layers with Hybrid Attention10 May 2026
Similar, not cited
Close in wording, no citation either way
- 2026MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
- 2026Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing
- 2026When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
- 2026Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
- 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models
- 2026Path-Constrained Mixture-of-Experts
- 2026Mixture of Experts for Low-Resource LLMs
- 2025Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing