mlsanityData to 7 Oct 2026

On the Spatial Structure of Mixture-of-Experts in Transformers

arXiv 2504.04444 ↗First posted 6 Apr 2025cs.CL cs.AI cs.LGcited by 2

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

A common assumption is that MoE routers primarily leverage semantic features for expert selection. However, our study challenges this notion by demonstrating that positional token information also plays a crucial role in routing decisions. Through extensive empirical analysis, we provide evidence supporting this hypothesis, develop a phenomenological explanation of the observed behavior, and discuss practical implications for MoE-based architectures.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 16 of its 28 references that are in the map. First posted on arXiv 6 Apr 2025.

Builds on

16 of 28 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 15

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2024Mixtral of Expertscited by 1,785
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  7. 2018Self-Attention with Relative Position Representationscited by 1,112
  8. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  9. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  10. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  11. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  12. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
3 more
  1. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  2. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  3. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEscited by 2
  2. 2026Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatorycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimizing MoE Routers: Design, Implementation, and Evaluation in Transformer Models
  2. 2024Part-Of-Speech Sensitivity of Routers in Mixture of Experts Models
  3. 2026Path-Constrained Mixture-of-Experts
  4. 2026A theoretical model for task routing in mixture-of-expert transformers
  5. 2024Routers in Vision Mixture of Experts: An Empirical Study
  6. 2026Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation
  7. 2026Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap
  8. 2026The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise