# Part-Of-Speech Sensitivity of Routers in Mixture of Experts Models

arXiv: https://arxiv.org/abs/2412.16971 · HTML: https://mlsanity.com/paper/2412.16971/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 8 of its 14 references that are in the map.
- First posted on arXiv 22 Dec 2024.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 8 of 8)

- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [ST-MoE: Designing Stable and Transferable Sparse Expert Models](https://mlsanity.com/paper/2202.08906/) (2022)
- [OLMoE: Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2409.02060/) (2024)
- [Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models](https://mlsanity.com/paper/2208.03306/) (2022)
- [OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2402.01739/) (2024)
- [A Review of Sparse Expert Models in Deep Learning](https://mlsanity.com/paper/2209.01667/) (2022)

## Built on by (most cited first, 7 of 7)

- [Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2507.11181/) (2025)
- [Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs](https://mlsanity.com/paper/2508.19594/) (2025)
- [Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs](https://mlsanity.com/paper/2604.17837/) (2026)
- [Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing](https://mlsanity.com/paper/2607.28308/) (2026)
- [DBES: A Systematic Benchmark and Metric Suite for Evaluating Expert Specialization in Large-Scale MoEs](https://mlsanity.com/paper/2605.18498/) (2026)
- [Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Experts](https://mlsanity.com/paper/2605.20610/) (2026)
- [Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation](https://mlsanity.com/paper/2610.02123/) (2026)

## Abstract (arXiv)

This study investigates the behavior of model-integrated routers in Mixture of Experts (MoE) models, focusing on how tokens are routed based on their linguistic features, specifically Part-of-Speech (POS) tags. The goal is to explore across different MoE architectures whether experts specialize in processing tokens with similar linguistic traits. By analyzing token trajectories across experts and layers, we aim to uncover how MoE models handle linguistic information. Findings from six popular MoE models reveal expert specialization for specific POS categories, with routing paths showing high predictive accuracy for POS, highlighting the value of routing paths in characterizing tokens.