# Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering

arXiv: https://arxiv.org/abs/2601.14050 · HTML: https://mlsanity.com/paper/2601.14050/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 33 of its 45 references that are in the map.
- First posted on arXiv 20 Jan 2026.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 30 of 33)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [How do Large Language Models Handle Multilingualism?](https://mlsanity.com/paper/2402.18815/) (2024)
- [Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages](https://mlsanity.com/paper/2310.14799/) (2023)
- [On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons](https://mlsanity.com/paper/2404.02431/) (2024)
- [Do Multilingual LLMs Think In English?](https://mlsanity.com/paper/2502.15603/) (2025)
- [The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities](https://mlsanity.com/paper/2411.04986/) (2024)
- [Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languages](https://mlsanity.com/paper/2501.06346/) (2025)
- [PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts](https://mlsanity.com/paper/2504.18428/) (2025)
- [Crosslingual Reasoning through Test-Time Scaling](https://mlsanity.com/paper/2505.05408/) (2025)
- [Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap](https://mlsanity.com/paper/2501.02448/) (2025)
- [Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoning](https://mlsanity.com/paper/2502.17407/) (2025)
- [Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages](https://mlsanity.com/paper/2402.12204/) (2024)
- [SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languages](https://mlsanity.com/paper/2407.19672/) (2024)
- [Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMs](https://mlsanity.com/paper/2406.09265/) (2024)
- [Multilingual Routing in Mixture-of-Experts](https://mlsanity.com/paper/2510.04694/) (2025)
- [The Emergence of Abstract Thought in Large Language Models Beyond Any Language](https://mlsanity.com/paper/2506.09890/) (2025)
- [AutoCAP: Towards Automatic Cross-lingual Alignment Planning for Zero-shot Chain-of-Thought](https://mlsanity.com/paper/2406.13940/) (2024)
- [SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignment](https://mlsanity.com/paper/2501.03681/) (2025)
- [Contributions of Transformer Attention Heads in Multi- and Cross-lingual Tasks](https://mlsanity.com/paper/2108.08375/) (2021)
- [Babel: Open Multilingual Large Language Models Serving Over 90% of Global Speakers](https://mlsanity.com/paper/2503.00865/) (2025)
- [Could Thinking Multilingually Empower LLM Reasoning?](https://mlsanity.com/paper/2504.11833/) (2025)
- [Enhancing Non-English Capabilities of English-Centric Large Language Models through Deep Supervision Fine-Tuning](https://mlsanity.com/paper/2503.01275/) (2025)
- [LayAlign: Enhancing Multilingual Reasoning in Large Language Models via Layer-Wise Adaptive Fusion and Alignment Strategy](https://mlsanity.com/paper/2502.11405/) (2025)
- [Disparities in LLM Reasoning Accuracy and Explanations: A Case Study on African American English](https://mlsanity.com/paper/2503.04099/) (2025)

## Built on by (most cited first, 10 of 10)

- [The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise](https://mlsanity.com/paper/2604.09780/) (2026)
- [AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning](https://mlsanity.com/paper/2608.05987/) (2026)
- [Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation](https://mlsanity.com/paper/2604.03592/) (2026)
- [RASA: Routing-Aware Safety Alignment for Mixture-of-Experts Models](https://mlsanity.com/paper/2602.04448/) (2026)
- [DiMtextsuperscript3: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Merging](https://mlsanity.com/paper/2605.12960/) (2026)
- [Code-Switching Reveals Language Anchoring in Multilingual LLMs](https://mlsanity.com/paper/2606.19668/) (2026)
- [Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models](https://mlsanity.com/paper/2605.28306/) (2026)
- [Is MoE Routing a Huffman Code? Discovering the Frequency-Diversity Law in Chain-of-Thought](https://mlsanity.com/paper/2607.20427/) (2026)
- [Travel Back To Move Forward: Retrospective Exploration and Evolving for Long Horizon Agents](https://mlsanity.com/paper/2610.13058/) (2026)
- [Overcoming Data Imbalances with Expert Selection Biases](https://mlsanity.com/paper/2610.13370/) (2026)

## Abstract (arXiv)

Mixture-of-Experts (MoE) architectures have shown strong multilingual capabilities, yet the internal mechanisms underlying performance gains and cross-language differences remain insufficiently understood. In this work, we conduct a systematic analysis of MoE models, examining routing behavior and expert specialization across languages and network depth. Our analysis reveals that multilingual processing in MoE models is highly structured: routing aligns with linguistic families, expert utilization follows a clear layerwise pattern, and high-resource languages rely on shared experts while low-resource languages depend more on language-exclusive experts despite weaker performance. Layerwise interventions further show that early and late MoE layers support language-specific processing, whereas middle layers serve as language-agnostic capacity hubs. Building on these insights, we propose a routing-guided steering method that adaptively guides routing behavior in middle layers toward shared experts associated with dominant languages at inference time, leading to consistent multilingual performance improvements, particularly for linguistically related language pairs. Our code is available at https://github.com/conctsai/Multilingualism-in-Mixture-of-Experts-LLMs.