# Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality

arXiv: https://arxiv.org/abs/2604.14419 · HTML: https://mlsanity.com/paper/2604.14419/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 32 of its 38 references that are in the map.
- First posted on arXiv 15 Apr 2026.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 30 of 32)

- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [ST-MoE: Designing Stable and Transferable Sparse Expert Models](https://mlsanity.com/paper/2202.08906/) (2022)
- [OLMoE: Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2409.02060/) (2024)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [Mixture-of-Depths: Dynamically allocating compute in transformer-based language models](https://mlsanity.com/paper/2404.02258/) (2024)
- [Hash Layers For Large Sparse Models](https://mlsanity.com/paper/2106.04426/) (2021)
- [Scaling Laws for Fine-Grained Mixture of Experts](https://mlsanity.com/paper/2402.07871/) (2024)
- [DEMix Layers: Disentangling Domains for Modular Language Modeling](https://mlsanity.com/paper/2108.05036/) (2021)
- [Mixture of A Million Experts](https://mlsanity.com/paper/2407.04153/) (2024)
- [ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing](https://mlsanity.com/paper/2412.14711/) (2024)
- [Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models](https://mlsanity.com/paper/2507.17702/) (2025)
- [Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidance](https://mlsanity.com/paper/2510.24711/) (2025)
- [Layerwise Recurrent Router for Mixture-of-Experts](https://mlsanity.com/paper/2408.06793/) (2024)
- [Statistical Advantages of Perturbing Cosine Router in Mixture of Experts](https://mlsanity.com/paper/2405.14131/) (2024)
- [Towards a Comprehensive Scaling Law of Mixture-of-Experts](https://mlsanity.com/paper/2509.23678/) (2025)
- [On Linear Mode Connectivity of Mixture-of-Experts Architectures](https://mlsanity.com/paper/2509.11348/) (2025)
- [Routing Manifold Alignment Improves Generalization of Mixture-of-Experts LLMs](https://mlsanity.com/paper/2511.07419/) (2025)
- [The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models](https://mlsanity.com/paper/2601.03425/) (2026)
- [MoE Lens -- An Expert Is All You Need](https://mlsanity.com/paper/2603.05806/) (2026)
- [BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference](https://mlsanity.com/paper/2511.10054/) (2025)
- [Grouter: Decoupling Routing from Representation for Accelerated MoE Training](https://mlsanity.com/paper/2603.06626/) (2026)

## Built on by (most cited first, 1 of 1)

- [Mixture of Layers with Hybrid Attention](https://mlsanity.com/paper/2605.09516/) (2026)

## Abstract (arXiv)

Sparse Mixture-of-Experts (MoE) architectures employ increasingly sophisticated routing mechanisms -- learned routers, multi-hop trajectories, token-dependent gating. We ask: does routing topology actually determine language modeling quality? We build a geometric MoE (ST-MoE) using cosine-similarity routing against learned centroids in a low-dimensional space (dspace = 64), requiring 80% fewer routing parameters than standard linear routers. Through 62 controlled experiments on WikiText-103 at 76--84M parameters trained to convergence (50K steps, 1.64B tokens), we find that routing topology does not determine asymptotic perplexity (PPL): five cosine-routing variants are statistically equivalent within a 1-PPL margin (Two One-Sided Tests [TOST], p < 0.05 for all 10 pairwise comparisons; 15 runs across 3 seeds, observed range 33.93--34.72). The finding extends to hash, random-fixed, and top-1 routing (single-seed; graceful 1.1--2.2 PPL degradation) and replicates on OpenWebText (0.03 PPL gap, 6 runs, 3 seeds each). A standard linear router with 5.3× more routing parameters reaches PPL 32.76, but iso-parameter cosine routing closes 67% of this gap -- the true mechanism advantage is ~1.2%. The mechanistic explanation is convergent redundancy: multi-hop updates are collinear (cos(Delta h0, Delta h1) = 0.805), implementing magnitude amplification rather than compositional reasoning; a single learnable scalar replicates multi-hop performance. As a practical payoff, zero-shot relative-norm halting saves 25% of MoE FLOPs at +0.12% PPL. Expert-level specialization and causal controllability -- which coexist with topology-level equifinality -- are explored in a companion paper.