# Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs

arXiv: https://arxiv.org/abs/2604.17837 · HTML: https://mlsanity.com/paper/2604.17837/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 9 of its 11 references that are in the map.
- First posted on arXiv 20 Apr 2026.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 9 of 9)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2402.01739/) (2024)
- [Granite Code Models: A Family of Open Foundation Models for Code Intelligence](https://mlsanity.com/paper/2405.04324/) (2024)
- [Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free](https://mlsanity.com/paper/2410.10814/) (2024)
- [Probing Semantic Routing in Large Mixture-of-Expert Models](https://mlsanity.com/paper/2502.10928/) (2025)
- [Part-Of-Speech Sensitivity of Routers in Mixture of Experts Models](https://mlsanity.com/paper/2412.16971/) (2024)
- [On the Spatial Structure of Mixture-of-Experts in Transformers](https://mlsanity.com/paper/2504.04444/) (2025)

## Built on by (most cited first, 2 of 2)

- [Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts](https://mlsanity.com/paper/2609.02404/) (2026)
- [Routing Divergence Is Not Evidence of Behavioral Influence in Same-Weight MoE Self-Distillation](https://mlsanity.com/paper/2608.15787/) (2026)

## Abstract (arXiv)

An LLM's residual stream is both state and instruction: it encodes the current context and determines the next transformation. We introduce a parameter-free decomposition for Mixture-of-Experts models that splits each layer's hidden state into a control signal that causally drives routing and an orthogonal content channel invisible to the router. Across six MoE architectures, we find that models preserve surface-level features (language, token identity, position) in the content channel, while the control signal encodes an abstract function that rotates from layer to layer. Because each routing decision is low-bandwidth, this hand-off forces compositional specialization across layers. While individual experts remain polysemantic, expert paths become monosemantic, clustering tokens by semantic function across languages and surface forms. The same token (e.g., ":") follows distinct trajectories depending on whether it serves as a type annotation, an introductory colon, or a time separator. Our decomposition identifies the source of this structure: clusters in the control subspace are substantially more monosemantic than those in the full representation. As a result, the natural unit of interpretability in MoEs is not the expert but the trajectory.