mlsanityData to 14 Oct 2026

Probing Semantic Routing in Large Mixture-of-Expert Models

arXiv 2502.10928 ↗First posted 15 Feb 2025cs.LG cs.AI cs.CLcited by 14

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

In the past year, large (>100B parameter) mixture-of-expert (MoE) models have become increasingly common in the open domain. While their advantages are often framed in terms of efficiency, prior work has also explored functional differentiation through routing behavior. We investigate whether expert routing in large MoE models is influenced by the semantics of the inputs. To test this, we design two controlled experiments. First, we compare activations on sentence pairs with a shared target word used in the same or different senses. Second, we fix context and substitute the target word with semantically similar or dissimilar alternatives. Comparing expert overlap across these conditions reveals clear, statistically significant evidence of semantic routing in large MoE models.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 13 of its 20 references that are in the map. First posted on arXiv 15 Feb 2025.

Builds on

13 of 20 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Towards an empirical understanding of MoE design choices20 Feb
  2. 2021Taming Sparsely Activated Transformer with Stochastic Experts8 Oct
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 10

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  3. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  4. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  5. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  6. 2021Hash Layers For Large Sparse Modelscited by 237
  7. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  8. 2024DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agentscited by 100
  9. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 48
  10. 2021Swords: A Benchmark for Lexical Substitution with Improved Data Coverage and Qualitycited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 13.

14 papers.

Sort:

By area:

  1. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  2. 2026The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Levelcited by 7
  3. 2025Meta-R1: Empowering Large Reasoning Models with Metacognitioncited by 6
  4. 2026The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Modelscited by 6
  5. 2026Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistencycited by 6
  6. 2026Extracting Small Translation Specialists from LLMs by Aggressively Pruning Expertscited by 3
  7. 2026Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEscited by 2
  8. 2026CoGR-MoE: Concept-Guided Expert Routing with Consistent Selection and Flexible Reasoning for Visual Question Answeringcited by 1
  9. 2026Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routingcited by 1
  10. 2026Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Expertscited by 0
  11. 2026Decomposing Refusal Steering in Mixture-of-Experts Modelscited by 0
  12. 2026Disagree to Explore, Agree to Commit: Routing-Guided Test-Time Scaling for Software Agentscited by 0
2 more
  1. 2026MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookupcited by 0
  2. 2026Not All Drift Is a Mechanism: Tracing Prompt Adaptation in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Routing by Analogy: kNN-Augmented Expert Assignment for Mixture-of-Experts
  2. 2026The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise
  3. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  4. 2026Geometric Routing Enables Causal Expert Control in Mixture of Experts
  5. 2024Routing in Sparsely-gated Language Models responds to Context
  6. 2026MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
  7. 2025Bayesian Mixture-of-Experts: Towards Making LLMs Know What They Don't Know
  8. 2025How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Experts