# One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Merging

arXiv: https://arxiv.org/abs/2604.02881 · HTML: https://mlsanity.com/paper/2604.02881/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 69 references that are in the map.
- First posted on arXiv 3 Apr 2026.

## Builds on (most cited first, 30 of 50)

- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Arcee's MergeKit: A Toolkit for Merging Large Language Models](https://mlsanity.com/paper/2403.13257/) (2024)
- [Task Singular Vectors: Reducing Task Interference in Model Merging](https://mlsanity.com/paper/2412.00081/) (2024)
- [Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages](https://mlsanity.com/paper/2104.05596/) (2021)
- [Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models](https://mlsanity.com/paper/2402.16438/) (2024)
- [EMR-Merging: Tuning-Free High-Performance Model Merging](https://mlsanity.com/paper/2405.17461/) (2024)
- [Neurons in Large Language Models: Dead, N-gram, Positional](https://mlsanity.com/paper/2309.04827/) (2023)
- [Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca](https://mlsanity.com/paper/2309.08958/) (2023)
- [FuseChat: Knowledge Fusion of Chat Models](https://mlsanity.com/paper/2408.07990/) (2024)
- [AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Merging](https://mlsanity.com/paper/2402.18913/) (2024)
- [Parameter Sharing Methods for Multilingual Self-Attentional Translation Models](https://mlsanity.com/paper/1809.00252/) (2018)

## Built on by (most cited first, 2 of 2)

- [One Model to Translate Them All: Universal Any-to-Any Translation for Heterogeneous Collaborative Perception](https://mlsanity.com/paper/2605.17907/) (2026)
- [On the Limits of Model Merging for Multilinguality in Pre-Training](https://mlsanity.com/paper/2605.25846/) (2026)

## Abstract (arXiv)

Weight-space model merging combines independently fine-tuned checkpoints without access to the original training data. While merging has shown promise in multitask settings, its behavior in multilingual generative systems remains underexplored. We systematically study weight-space merging for multilingual machine translation by fully fine-tuning language models on large-scale bilingual corpora and evaluating representative merging strategies across shared-source, shared-target, and bidirectional consolidation settings. Our experiments reveal a strong directional asymmetry. Merging is comparatively more effective when models share a target language, improving multilingual coverage over the base model, but it still fails to preserve the peak performance of language-specific checkpoints. In contrast, when target languages differ, performance degrades sharply, especially in shared-source and bidirectional settings. To explain this behavior, we analyze internal representations and find that fine-tuning does not create disjoint language-specific sub-networks. Instead, independently fine-tuned models activate largely overlapping neurons while reshaping upper-layer target-generation representations into incompatible geometries. These findings suggest that multilingual merging failures arise from target-side geometric misalignment within shared computational units, challenging the assumptions underlying standard weight-space merging for multilingual translation. We make the code publicly available at https://github.com/babangain/mt-model-merging