mlsanityData to 14 Oct 2026

Part-Of-Speech Sensitivity of Routers in Mixture of Experts Models

arXiv 2412.16971 ↗First posted 22 Dec 2024cs.CLcited by 7

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

This study investigates the behavior of model-integrated routers in Mixture of Experts (MoE) models, focusing on how tokens are routed based on their linguistic features, specifically Part-of-Speech (POS) tags. The goal is to explore across different MoE architectures whether experts specialize in processing tokens with similar linguistic traits. By analyzing token trajectories across experts and layers, we aim to uncover how MoE models handle linguistic information. Findings from six popular MoE models reveal expert specialization for specific POS categories, with routing paths showing high predictive accuracy for POS, highlighting the value of routing paths in characterizing tokens.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 8 of its 14 references that are in the map. First posted on arXiv 22 Dec 2024.

Builds on

8 of 14 references.

References · 8

  1. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  2. 2024Mixtral of Expertscited by 1,823
  3. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  4. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  5. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  6. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  7. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  8. 2022A Review of Sparse Expert Models in Deep Learningcited by 148

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025Mixture of Experts in Large Language Modelscited by 15
  2. 2025Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMscited by 12
  3. 2026Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEscited by 2
  4. 2026Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routingcited by 1
  5. 2026DBES: A Systematic Benchmark and Metric Suite for Evaluating Expert Specialization in Large-Scale MoEscited by 0
  6. 2026Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Expertscited by 0
  7. 2026Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
  2. 2026Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality
  3. 2025Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognition
  4. 2026Safety-Oriented Routing Analysis of Mixtral MoE Under Benign and Harmful Prompts
  5. 2025Multilingual Routing in Mixture-of-Experts
  6. 2024DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models
  7. 2026When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
  8. 2025Bayesian Mixture-of-Experts: Towards Making LLMs Know What They Don't Know