# Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts

arXiv: https://arxiv.org/abs/2609.02404 · HTML: https://mlsanity.com/paper/2609.02404/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 14 of its 14 references that are in the map.
- First posted on arXiv 2 Sep 2026.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 14 of 14)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [OLMoE: Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2409.02060/) (2024)
- [OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2402.01739/) (2024)
- [Generalized Shape Metrics on Neural Representations](https://mlsanity.com/paper/2110.14739/) (2021)
- [Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gate](https://mlsanity.com/paper/2502.12224/) (2025)
- [When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2605.07260/) (2026)
- [Multi-Way Representation Alignment](https://mlsanity.com/paper/2602.06205/) (2026)
- [Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts](https://mlsanity.com/paper/2605.12476/) (2026)
- [A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference](https://mlsanity.com/paper/2606.15453/) (2026)
- [Representing Neural Network Layers as Linear Operations via Koopman Operator Theory](https://mlsanity.com/paper/2409.01308/) (2024)
- [Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs](https://mlsanity.com/paper/2604.17837/) (2026)

## Built on by (most cited first, 2 of 2)

- [Beyond the Previous Layer: Residual Predictive Structure in Sparse MoE Routing](https://mlsanity.com/paper/2609.17940/) (2026)
- [HERO-MoE: Historical Expert Routing with Scale-Preserving Fusion](https://mlsanity.com/paper/2609.32581/) (2026)

## Abstract (arXiv)

Sparse mixture-of-experts (MoE) models use an independently parameterized router at each sparse layer to select experts for every token. Prior work has shown that routing decisions across depth can often be predicted from earlier routing signals, suggesting that routing is not fully independent across layers. However, the structure behind this predictability remains unclear. In this work, we provide evidence that routing-relevant states across layers share a common geometric structure that is obscured by layer-specific coordinate systems. We isolate the control subspace of each router and align these spaces into a shared canonical representation using generalized orthogonal Procrustes analysis. After alignment, a single linear transition reaches R2=0.39--0.71 and retains 79--90% of the predictive power of separately fitted layer-specific dynamics, indicating that much of routing-state evolution follows a reusable process across depth. We then ask whether this shared dynamics is specific to routing or simply reflects the smooth evolution of hidden representations. A matched-rank comparison shows that residual representations are often easier to predict across layers, while router-control states preserve the model's expert choices much more faithfully. This separates generic cross-layer predictability from routing-specific information. Finally, we test whether the predicted canonical states remain meaningful when used in place of native routing states. The transported states preserve local routing behavior, while learned state evolution reduces DeltaNLL relative to simple persistence by 15.7% on OLMoE and 6.2% over a 10-router horizon on Phi.