# Multilingual Routing in Mixture-of-Experts

arXiv: https://arxiv.org/abs/2510.04694 · HTML: https://mlsanity.com/paper/2510.04694/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 45 of its 52 references that are in the map.
- First posted on arXiv 6 Oct 2025.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 30 of 45)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [OLMoE: Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2409.02060/) (2024)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints](https://mlsanity.com/paper/2212.05055/) (2022)
- [Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation](https://mlsanity.com/paper/2412.03304/) (2024)
- [OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2402.01739/) (2024)
- [Efficient Large Scale Language Modeling with Mixtures of Experts](https://mlsanity.com/paper/2112.10684/) (2021)
- [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://mlsanity.com/paper/2205.06266/) (2022)
- [Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models](https://mlsanity.com/paper/2402.16438/) (2024)
- [Large Concept Models: Language Modeling in a Sentence Representation Space](https://mlsanity.com/paper/2412.08821/) (2024)
- [On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons](https://mlsanity.com/paper/2404.02431/) (2024)
- [The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities](https://mlsanity.com/paper/2411.04986/) (2024)
- [ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora](https://mlsanity.com/paper/2012.15674/) (2020)
- [Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations](https://mlsanity.com/paper/2310.20246/) (2023)
- [Learning Joint Multilingual Sentence Representations with Neural Machine Translation](https://mlsanity.com/paper/1704.04154/) (2017)
- [A Closer Look into Mixture-of-Experts in Large Language Models](https://mlsanity.com/paper/2406.18219/) (2024)
- [Crosslingual Reasoning through Test-Time Scaling](https://mlsanity.com/paper/2505.05408/) (2025)
- [PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuning](https://mlsanity.com/paper/2311.08711/) (2023)
- [Steering MoE LLMs via Expert (De)Activation](https://mlsanity.com/paper/2509.09660/) (2025)

## Built on by (most cited first, 29 of 29)

- [The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise](https://mlsanity.com/paper/2604.09780/) (2026)
- [Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering](https://mlsanity.com/paper/2601.14050/) (2026)
- [Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts](https://mlsanity.com/paper/2604.08541/) (2026)
- [Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistency](https://mlsanity.com/paper/2603.17102/) (2026)
- [Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation](https://mlsanity.com/paper/2604.03592/) (2026)
- [Towards Intrinsic Interpretability of Large Language Models:A Survey of Design Principles and Architectures](https://mlsanity.com/paper/2604.16042/) (2026)
- [What Gets Activated: Uncovering Domain and Driver Experts in MoE Language Models](https://mlsanity.com/paper/2601.10159/) (2026)
- [Extracting Small Translation Specialists from LLMs by Aggressively Pruning Experts](https://mlsanity.com/paper/2605.28042/) (2026)
- [Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation](https://mlsanity.com/paper/2605.29714/) (2026)
- [SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignment](https://mlsanity.com/paper/2606.25821/) (2026)
- [Multilinguality at the Edge: Developing Language Models for the Global South](https://mlsanity.com/paper/2604.21637/) (2026)
- [HyDRA: Hybrid Dynamic Routing Architecture for Heterogeneous LLM Pools](https://mlsanity.com/paper/2605.17106/) (2026)
- [Can LLMs Cook Jamaican Couscous? A Study of Cultural Novelty in Recipe Generation](https://mlsanity.com/paper/2602.10964/) (2026)
- [A Closer Look into LLMs for Table Understanding](https://mlsanity.com/paper/2603.15402/) (2026)
- [Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory](https://mlsanity.com/paper/2605.20982/) (2026)
- [Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models](https://mlsanity.com/paper/2605.28306/) (2026)
- [Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models](https://mlsanity.com/paper/2606.03780/) (2026)
- [Multilingual Fine-Tuning via Localized Gradient Conflict Resolution](https://mlsanity.com/paper/2606.05613/) (2026)
- [How Modular Is a Frontier Mixture-of-Experts? A Pre-registered Causal Test in Which Apparent Expert Modularity Mostly Dissolves](https://mlsanity.com/paper/2606.25092/) (2026)
- [Lightweight Chunk Selection for Mobile Retrieval-Augmented Generation](https://mlsanity.com/paper/2608.03148/) (2026)
- [ESTS at WMT26: Routing-Informed Expert Pruning for Model Compression](https://mlsanity.com/paper/2609.12310/) (2026)
- [A Persistent State for Auditable Mixture-of-Experts Routing](https://mlsanity.com/paper/2609.34634/) (2026)
- [Multilinguality in Hybrid Attention LLMs](https://mlsanity.com/paper/2609.35378/) (2026)
- [From Routing Signals to Selective Review: Visual regrounding in MoE VLMs](https://mlsanity.com/paper/2609.38111/) (2026)
- [Cross-Lingual Alignment for Decoder-Only Models using MoE Routers](https://mlsanity.com/paper/2610.01921/) (2026)
- [Frequency Is Not Sensitivity Identifying Safety-Sensitive Experts in Sparse MoE LLM](https://mlsanity.com/paper/2610.02910/) (2026)
- [Overcoming Data Imbalances with Expert Selection Biases](https://mlsanity.com/paper/2610.13370/) (2026)
- [Scaling Laws for Multimodal Data Mixtures](https://mlsanity.com/paper/2610.14423/) (2026)
- [Towards Robust Calibration Dataset for Domain-General Mixture of Experts Pruning](https://mlsanity.com/paper/2610.15578/) (2026)

## Abstract (arXiv)

Mixture-of-Experts (MoE) architectures have become the key to scaling modern LLMs, yet little is understood about how their sparse routing dynamics respond to multilingual data. In this work, we analyze expert routing patterns using parallel multilingual datasets and present highly interpretable layer-wise phenomena. We find that MoE models route tokens in language-specific ways in the early and late decoder layers but exhibit significant cross-lingual routing alignment in middle layers, mirroring parameter-sharing trends observed in dense LLMs. In particular, we reveal a clear, strong correlation between a model's performance in a given language and how similarly its tokens are routed to English in these layers. Extending beyond correlation, we explore inference-time interventions that induce higher cross-lingual routing alignment. We introduce a method that steers the router by promoting middle-layer task experts frequently activated in English, and it successfully increases multilingual performance. These 1-2% gains are remarkably consistent across two evaluation tasks, three models, and 15+ languages, especially given that these simple interventions override routers of extensively trained, state-of-the-art LLMs. In comparison, interventions outside of the middle layers or targeting multilingual-specialized experts only yield performance degradation. Altogether, we present numerous findings that explain how MoEs process non-English text and demonstrate that generalization is limited by the model's ability to leverage language-universal experts in all languages.