mlsanityData to 14 Oct 2026

A Declarative-Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models

arXiv 2608.15102 ↗First posted 15 Aug 2026cs.CLcited by 0

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

We investigate whether Mixture-of-Experts (MoE) language models develop linguistically structured expert routing during bilingual language acquisition. Inspired by the Declarative-Procedural framework, we analyze lexical, grammatical, and syntactic processing in a decoder-only English-German MoE Transformer trained under sequential language exposure. We construct a probe-based validation set and extract token-level routing distributions to quantify category-dependent specialisation using mutual information, routing entropy, and Jensen-Shannon distance. The curriculum-trained model exhibits a peak mutual information of 0.1148 at layer 5, indicating category-dependent differences in routing distributions across linguistic categories. Surprisingly, a no-curriculum baseline trained on mixed English-German data shows stronger aggregate specialisation, reaching a peak mutual information of 0.2599 at the same layer. These results suggest that interpretable linguistic organization emerges within MoE routing patterns even without sequential language exposure. A replication at a second training seed shows that the no-curriculum condition's specialisation concentrates on a single language whose identity is seed-dependent, whereas the curriculum consistently yields a stable, language-balanced routing profile; rather than uniformly increasing specialisation, staged bilingual exposure reduces single-language dominance. The official Github repository: https://github.com/Amrit828/DP-Theory-MOE-Interpretability-Research

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 10 of its 20 references that are in the map. First posted on arXiv 15 Aug 2026.

Builds on

10 of 20 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale8 Apr
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 8

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  5. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  6. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  8. 2019Competence-based Curriculum Learning for Neural Machine Translationcited by 202

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering
  2. 2026Mixture of Experts for Low-Resource LLMs
  3. 2026Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation
  4. 2025Multilingual Routing in Mixture-of-Experts
  5. 2026Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models
  6. 2026Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation
  7. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  8. 2025Probing Semantic Routing in Large Mixture-of-Expert Models