mlsanityData to 14 Oct 2026

Cosine-Similarity Routing with Semantic Anchors for Interpretable Mixture-of-Experts Language Models

arXiv 2509.14255 ↗First posted 12 Sep 2025cs.CL cs.AIcited by 0

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Mixture-of-Experts (MoE) models improve efficiency through sparse activation, but their learned gating functions provide limited insight into routing decisions. This work introduces the Semantic Resonance Architecture (SRA), which routes tokens to experts via cosine similarity between token representations and learnable semantic anchors, making every routing decision directly traceable to anchor-token similarity scores. We evaluate SRA on WikiText-103 across 17 configurations. In a controlled multi-seed comparison (3 seeds x 4 configurations, 256 experts, Dff=256), cosine routing achieves competitive perplexity with standard linear routing (12.57 ± 0.03 vs 12.45 ± 0.03 for K=1 → 4; 12.52 ± 0.02 vs 12.57 ± 0.02 for K=2 → 4). The training recipe -- not the routing function -- drives specialization quality, while cosine routing provides inherent inspectability. We introduce a bandpass routing loss -- a floor-and-ceiling corridor on expert utilization -- that reduces dead experts from 30-45% to 0-6% and transfers to both routing types. Routing-space evaluation shows cosine routing provides significantly better word-level subtoken coherence in deeper layers (p < 0.001), with 44-54% of expert specialization being syntactic rather than semantic. Extended analysis reveals cosine routing maintains more stable router saturation and tighter per-expert vocabulary distributions -- structural advantages from the bounded cosine similarity range. An inference-time k-sweep shows that k=5 yields a free 0.08-0.16 perplexity gain over k=4. Cross-dataset validation on OpenWebText confirms generalization: cosine routing achieves comparable perplexity (44.88 vs 45.44), the bandpass loss eliminates dead experts, and specialization patterns are preserved.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 29 of its 41 references that are in the map. First posted on arXiv 12 Sep 2025.

Builds on

29 of 41 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing19 Dec
  2. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Experts23 May
  3. 2022On the Representation Collapse of Sparse Mixture of Experts20 Apr
  4. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  10. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  11. 2016Pointer Sentinel Mixture Modelscited by 3,096
  12. 2018Deep contextualized word representationscited by 2,290
12 more
  1. 2024Mixtral of Expertscited by 1,823
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  3. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  4. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  5. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  6. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  7. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  8. 2023From Sparse to Soft Mixtures of Expertscited by 248
  9. 2021Hash Layers For Large Sparse Modelscited by 237
  10. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  11. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 132
  12. 2021AI Explainability 360: Impact and Designcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality
  2. 2026Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts
  3. 2026Mixture of Experts for Low-Resource LLMs
  4. 2025Probing Semantic Routing in Large Mixture-of-Expert Models
  5. 2024Expert-Token Resonance MoE: Bidirectional Routing with Efficiency Affinity-Driven Active Selection
  6. 2026Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing
  7. 2026RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Models
  8. 2026Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models