mlsanityData to 7 Oct 2026

SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignment

arXiv 2606.25821 ↗First posted 24 Jun 2026cs.CL cs.AIcited by 0

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Sparse Mixture-of-Experts (MoE) architectures have emerged as an increasingly influential paradigm as they offer a strategic balance between parameter scalability and computational efficiency. However, low-resource languages, which suffer from a scarcity of high-quality training data, often have their tokens routed to different experts than those predominantly activated by high-resource inputs, which limits cross-lingual expert sharing. This cross-lingual routing divergence consequently hinders their efficacy in multilingual contexts. To address this issue, we propose SARA (Semantically Anchored Routing Alignment), a framework designed to transfer specialized capabilities from high-resource languages as anchors to low-resource languages. SARA explicitly aligns the routing distribution of multilingual inputs with high-resource semantic anchors using a symmetric Jensen-Shannon (JS) divergence constraint. Unlike traditional distillation methods that operate on output logits, SARA directly aligns the internal routing distributions of MoE layers, encouraging mechanistic consistency in expert selection across languages. We conduct experiments on 2 LLMs across 5 low-resource languages and 3 benchmarks. Experiment results demonstrate that SARA outperforms standard instruction tuning, e.g., +0.8% on Qwen3-30B-A3B and +1.2% on Phi-3.5-MoE-instruct on Global-MMLU. Further analyses show that SARA effectively addresses performance bottlenecks in low-resource languages, providing a scalable pathway to enhance multilingual capabilities in sparse architectures.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 19 of its 38 references that are in the map. First posted on arXiv 24 Jun 2026.

Builds on

19 of 38 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Routing Manifold Alignment Improves Generalization of Mixture-of-Experts LLMs10 Nov
  2. 2025Multilingual Routing in Mixture-of-Experts6 Oct
  3. 2022On the Representation Collapse of Sparse Mixture of Experts20 Apr
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 15

  1. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  2. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  3. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  4. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 181
  5. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  6. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 134
  7. 2023Do Multilingual Language Models Think Better in English?cited by 87
  8. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Expertscited by 36
  9. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selectioncited by 27
  10. 2024HyperMoE: Towards Better Mixture of Experts via Transferring Among Expertscited by 25
  11. 2024MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routingcited by 16
  12. 2024MIT-10M: A Large Scale Parallel Corpus of Multilingual Image Translationcited by 12
3 more
  1. 2025Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resourcescited by 10
  2. 2021Unsupervised Domain Adaptation Schemes for Building ASR in Low-resource Languagescited by 9
  3. 2023H-AES: Towards Automated Essay Scoring for Hindicited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering
  2. 2026Mixture of Experts for Low-Resource LLMs
  3. 2026Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation
  4. 2026Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation
  5. 2026Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models
  6. 2026Slicing and Dicing: Configuring Optimal Mixtures of Experts
  7. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models
  8. 2025How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Experts