mlsanityData to 14 Oct 2026

Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs

arXiv 2605.24681 ↗First posted 23 May 2026cs.CL cs.AIcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Large Language Models (LLMs) have shown great promise in multilingual machine translation (MT), even with limited bilingual supervision. However, fine-tuning LLMs with parallel corpora presents major challenges, namely parameter interference. To address these issues, we propose Mix-MoE, a mixed Mixture-of-Experts framework designed to train LLMs for multilingual MT. Our framework operates in two distinct stages: (1) post-pretraining with MoE on monolingual corpora, and (2) post-pretraining with MoE on parallel corpora. Crucially, we divide the MoE layers into two specialized groups: Language Model Experts (LM Experts) and Machine Translation Experts (MT Experts). LM Experts are designed to capture and retain the monolingual knowledge learned by the pre-trained LLM. MT Experts, on the other hand, are specifically trained to acquire and store bilingual translation knowledge. Furthermore, to facilitate effective interaction between these specialized experts and leverage potential underlying structural patterns in text, we introduce a routing mechanism enhanced by Fourier Transform features derived from model representations. The experimental results demonstrate that Mix-MoE excels in multilingual MT, significantly outperforming existing baselines and showing notable progress in mitigating parameter interference.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 46 of its 58 references that are in the map. First posted on arXiv 23 May 2026.

Builds on

46 of 58 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training24 Nov
  2. 2024OLMoE: Open Mixture-of-Experts Language Models3 Sep
  3. 2024MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing21 Aug
  4. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training24 Jun
  5. 2024Mixtral of Experts8 Jan
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translation11 Jul
  7. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec

Other references · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  7. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  8. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  9. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  10. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  11. 2023From Sparse to Soft Mixtures of Expertscited by 248
  12. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
27 more
  1. 2016Transfer Learning for Low-Resource Neural Machine Translationcited by 188
  2. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 160
  3. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  4. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 144
  5. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 127
  6. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 101
  7. 2020Overcoming Catastrophic Forgetting in Graph Neural Networkscited by 95
  8. 2021Learnable Fourier Features for Multi-Dimensional Spatial Positional Encodingcited by 93
  9. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 89
  10. 2024On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neuronscited by 83
  11. 2018Rapid Adaptation of Neural Machine Translation to New Languagescited by 82
  12. 2024Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationcited by 39
  13. 2024X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scalecited by 36
  14. 2023Efficient Continual Pre-training for Building Domain Specific Large Language Modelscited by 34
  15. 2021Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoderscited by 24
  16. 2023MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Expertscited by 21
  17. 2023A Paradigm Shift: The Future of Machine Translation Lies with Large Language Modelscited by 19
  18. 2021Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillationcited by 15
  19. 2024Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignmentcited by 14
  20. 2023Learning a Fourier Transform for Linear Relative Positional Encodings in Transformerscited by 10
  21. 2023Towards Higher Pareto Frontier in Multilingual Machine Translationcited by 9
  22. 2022Multilingual Mix: Example Interpolation Improves Multilingual Neural Machine Translationcited by 7
  23. 2024Towards Incremental Learning in Large Language Models: A Critical Reviewcited by 6
  24. 2019DNN-based cross-lingual voice conversion using Bottleneck Featurescited by 2
  25. 2024Exploiting Domain-Specific Parallel Data on Multilingual Language Models for Low-resource Language Translationcited by 2
  26. 2018Deep Learning Based Natural Language Processing for End to End Speech Translationcited by 1
  27. 2023Optimal Transport Aggregation for Distributed Mixture-of-Expertscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation
  2. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models
  3. 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation
  4. 2026A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAMDelta Integration into Upcycled MoE
  5. 2026Extracting Small Translation Specialists from LLMs by Aggressively Pruning Experts
  6. 2024Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages
  7. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
  8. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference