mlsanityData to 14 Oct 2026

Geometric Routing Enables Causal Expert Control in Mixture of Experts

arXiv 2604.14434 ↗First posted 15 Apr 2026cs.AIcited by 2

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Sparse Mixture-of-Experts (MoE) models scale parameters while fixing active computation per token, but the specialization of individual experts remains opaque. In a companion paper we showed that routing topology is quality-neutral: five structurally different configurations converge to statistically equivalent language modeling quality. Here we show that expert identity is nonetheless causally meaningful: individual rank-1 experts are monosemantic by construction, and cosine-similarity routing in a low-dimensional metric space makes their specialization directly inspectable. We present four lines of evidence. First, projecting expert output vectors through the unembedding matrix yields a Semantic Dictionary: 15% of experts are monosemantic specialists spanning 10 categories (temporal, geographic, cardinal, discourse, emotional, financial, military, scientific). Second, routing exhibits a frequency-to-syntax gradient: early layers separate tokens by word frequency, deeper layers by syntactic class (Zipf-confound controls, all p < 0.001). Third, causal interventions confirm these labels: steering toward a temporal expert's centroid increases P(temporal) by +321% (median across 44 prompts); suppressing a geographic expert drops P(geographic) by -23%; rewriting an expert's output vector halves target-category probability, and effects compose additively across layers. Fourth, the interventions are not unique to cosine routing: linear routers support comparable steering, but only cosine routing provides geometric transparency -- expert specialization is readable directly from the centroid matrix. MoE expert-level specialization is a first-class interpretability primitive: architecturally monosemantic, causally validated, and controllable at inference with zero overhead.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 25 of its 32 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

25 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models6 Jan
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 23

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2024Mixtral of Expertscited by 1,823
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  7. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  8. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  9. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  10. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  11. 2022Toy Models of Superpositioncited by 971
  12. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
11 more
  1. 2024Scaling and evaluating sparse autoencoderscited by 553
  2. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  3. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoderscited by 160
  4. 2025Sparse Autoencoders Do Not Find Canonical Units of Analysiscited by 88
  5. 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 78
  6. 2024Decomposing The Dark Matter of Sparse Autoencoderscited by 45
  7. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Modelscited by 45
  8. 2024Monet: Mixture of Monosemantic Experts for Transformerscited by 13
  9. 2025Towards a Comprehensive Scaling Law of Mixture-of-Expertscited by 10
  10. 2025Explaining Large Language Models with gSMILEcited by 4
  11. 2025Sparsity and Superposition in Mixture of Expertscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatorycited by 0
  2. 2026Score the Update, Not the Token: Descent-Aligned Routing for Combinatorial LoRA Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing
  2. 2025Probing Semantic Routing in Large Mixture-of-Expert Models
  3. 2026The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise
  4. 2026Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality
  5. 2024Routing in Sparsely-gated Language Models responds to Context
  6. 2026Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts
  7. 2026The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level
  8. 2026A Declarative-Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models