# The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise

arXiv: https://arxiv.org/abs/2604.09780 · HTML: https://mlsanity.com/paper/2604.09780/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 27 of its 32 references that are in the map.
- First posted on arXiv 10 Apr 2026.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 27 of 27)

- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [Layer by Layer: Uncovering Hidden Representations in Language Models](https://mlsanity.com/paper/2502.02013/) (2025)
- [OLMoE: Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2409.02060/) (2024)
- [MathArena: Evaluating LLMs on Uncontaminated Math Competitions](https://mlsanity.com/paper/2505.23281/) (2025)
- [Massive Activations in Large Language Models](https://mlsanity.com/paper/2402.17762/) (2024)
- [Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts](https://mlsanity.com/paper/2408.15664/) (2024)
- [OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2402.01739/) (2024)
- [The Remarkable Robustness of LLMs: Stages of Inference?](https://mlsanity.com/paper/2406.19384/) (2024)
- [Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse](https://mlsanity.com/paper/2206.03126/) (2022)
- [On the Impact of the Activation Function on Deep Neural Networks Training](https://mlsanity.com/paper/1902.06853/) (2019)
- [Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models](https://mlsanity.com/paper/2501.11873/) (2025)
- [Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset](https://mlsanity.com/paper/2508.15096/) (2025)
- [Multilingual Routing in Mixture-of-Experts](https://mlsanity.com/paper/2510.04694/) (2025)
- [Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs](https://mlsanity.com/paper/2503.05139/) (2025)
- [Load Balancing Mixture of Experts with Similarity Preserving Routers](https://mlsanity.com/paper/2506.14038/) (2025)
- [MoEcho: Exploiting Side-Channel Attacks to Compromise User Privacy in Mixture-of-Experts LLMs](https://mlsanity.com/paper/2508.15036/) (2025)
- [Arcee Trinity Large Technical Report](https://mlsanity.com/paper/2602.17004/) (2026)
- [Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering](https://mlsanity.com/paper/2601.14050/) (2026)
- [Differential syntactic and semantic encoding in LLMs](https://mlsanity.com/paper/2601.04765/) (2026)
- [The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models](https://mlsanity.com/paper/2601.03425/) (2026)
- [Expert Selections In MoE Models Reveal (Almost) As Much As Text](https://mlsanity.com/paper/2602.04105/) (2026)
- [MoE Lens -- An Expert Is All You Need](https://mlsanity.com/paper/2603.05806/) (2026)
- [Expert Divergence Learning for MoE-based Language Models](https://mlsanity.com/paper/2603.00054/) (2026)
- [What Gets Activated: Uncovering Domain and Driver Experts in MoE Language Models](https://mlsanity.com/paper/2601.10159/) (2026)

## Built on by (most cited first, 13 of 13)

- [EMO: Pretraining Mixture of Experts for Emergent Modularity](https://mlsanity.com/paper/2605.06663/) (2026)
- [Sparse Mixture-of-Experts Routing in Visual Diffusion Transformers:Diagnosis, Boundary Calibration and Evolutionary Roadmap from Routing Collapse to Selective Deadlock](https://mlsanity.com/paper/2605.19378/) (2026)
- [Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-training](https://mlsanity.com/paper/2606.11867/) (2026)
- [Does the Same Token Mean the Same State? MoE Routing as Signal for Reasoning Control](https://mlsanity.com/paper/2606.22798/) (2026)
- [Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing](https://mlsanity.com/paper/2607.28308/) (2026)
- [Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models](https://mlsanity.com/paper/2606.03780/) (2026)
- [MoE Router-Guided Clustering for Heterogeneous Federated Instruction Tuning](https://mlsanity.com/paper/2608.15311/) (2026)
- [From Concentration to Differentiation and Back: Routing Effective Rank in MoE Reasoning Cohorts](https://mlsanity.com/paper/2609.06403/) (2026)
- [Raw-Routed Mixture of Adapters: A Causal Intervention for Routing Collapse in Time Series Foundation Models](https://mlsanity.com/paper/2609.39445/) (2026)
- [From Task Mixtures to Specialized Experts](https://mlsanity.com/paper/2610.00580/) (2026)
- [Conditional Capacity and Routing in Mixture-of-Experts Particle Transformers](https://mlsanity.com/paper/2610.02701/) (2026)
- [Structuring MoE Expert Selection for Agentic Reinforcement Learning](https://mlsanity.com/paper/2610.07332/) (2026)
- [Overcoming Data Imbalances with Expert Selection Biases](https://mlsanity.com/paper/2610.13370/) (2026)

## Abstract (arXiv)

Mixture of Experts (MoEs) are now ubiquitous in large language models, yet the mechanisms behind their "expert specialization" remain poorly understood. We show that, since MoE routers are linear maps, hidden state similarity is both necessary and sufficient to explain expert usage similarity, and specialization is therefore an emergent property of the representation space, not of the routing architecture itself. We confirm this at both token and sequence level across five pre-trained models. We additionally prove that load-balancing loss suppresses shared hidden state directions to maintain routing diversity, which might provide a theoretical explanation for specialization collapse under less diverse data, e.g. small batch. Despite this clean mechanistic account, we find that specialization patterns in pre-trained MoEs resist human interpretation: expert overlap between different models answering the same question is no higher than between entirely different questions (~60%); prompt-level routing does not predict rollout-level routing; and deeper layers exhibit near-identical expert activation across semantically unrelated inputs, especially in reasoning models. We conclude that, while the efficiency perspective of MoEs is well understood, understanding expert specialization is at least as hard as understanding LLM hidden state geometry, a long-standing open problem in the literature.