# Probing Semantic Routing in Large Mixture-of-Expert Models

arXiv: https://arxiv.org/abs/2502.10928 · HTML: https://mlsanity.com/paper/2502.10928/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 13 of its 20 references that are in the map.
- First posted on arXiv 15 Feb 2025.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 13 of 13)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Jamba: A Hybrid Transformer-Mamba Language Model](https://mlsanity.com/paper/2403.19887/) (2024)
- [ST-MoE: Designing Stable and Transferable Sparse Expert Models](https://mlsanity.com/paper/2202.08906/) (2022)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [Hash Layers For Large Sparse Models](https://mlsanity.com/paper/2106.04426/) (2021)
- [OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2402.01739/) (2024)
- [Taming Sparsely Activated Transformer with Stochastic Experts](https://mlsanity.com/paper/2110.04260/) (2021)
- [DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents](https://mlsanity.com/paper/2406.06769/) (2024)
- [From Tokens to Words: On the Inner Lexicon of LLMs](https://mlsanity.com/paper/2410.05864/) (2024)
- [Towards an empirical understanding of MoE design choices](https://mlsanity.com/paper/2402.13089/) (2024)
- [Swords: A Benchmark for Lexical Substitution with Improved Data Coverage and Quality](https://mlsanity.com/paper/2106.04102/) (2021)

## Built on by (most cited first, 14 of 14)

- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level](https://mlsanity.com/paper/2604.02178/) (2026)
- [Meta-R1: Empowering Large Reasoning Models with Metacognition](https://mlsanity.com/paper/2508.17291/) (2025)
- [The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models](https://mlsanity.com/paper/2601.03425/) (2026)
- [Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistency](https://mlsanity.com/paper/2603.17102/) (2026)
- [Extracting Small Translation Specialists from LLMs by Aggressively Pruning Experts](https://mlsanity.com/paper/2605.28042/) (2026)
- [Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs](https://mlsanity.com/paper/2604.17837/) (2026)
- [CoGR-MoE: Concept-Guided Expert Routing with Consistent Selection and Flexible Reasoning for Visual Question Answering](https://mlsanity.com/paper/2604.16930/) (2026)
- [Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing](https://mlsanity.com/paper/2607.28308/) (2026)
- [Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Experts](https://mlsanity.com/paper/2605.20610/) (2026)
- [Decomposing Refusal Steering in Mixture-of-Experts Models](https://mlsanity.com/paper/2606.04160/) (2026)
- [Disagree to Explore, Agree to Commit: Routing-Guided Test-Time Scaling for Software Agents](https://mlsanity.com/paper/2608.22191/) (2026)
- [MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup](https://mlsanity.com/paper/2609.15126/) (2026)
- [Not All Drift Is a Mechanism: Tracing Prompt Adaptation in LLMs](https://mlsanity.com/paper/2610.18412/) (2026)

## Abstract (arXiv)

In the past year, large (>100B parameter) mixture-of-expert (MoE) models have become increasingly common in the open domain. While their advantages are often framed in terms of efficiency, prior work has also explored functional differentiation through routing behavior. We investigate whether expert routing in large MoE models is influenced by the semantics of the inputs. To test this, we design two controlled experiments. First, we compare activations on sentence pairs with a shared target word used in the same or different senses. Second, we fix context and substitute the target word with semantically similar or dissimilar alternatives. Comparing expert overlap across these conditions reveals clear, statistically significant evidence of semantic routing in large MoE models.