# Merge and Conquer: Instructing Multilingual Models by Adding Target Language Weights

arXiv: https://arxiv.org/abs/2603.28263 · HTML: https://mlsanity.com/paper/2603.28263/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 48 references that are in the map.
- First posted on arXiv 30 Mar 2026.

## Builds on (most cited first, 26 of 26)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [Evolutionary Optimization of Model Merging Recipes](https://mlsanity.com/paper/2403.13187/) (2024)
- [Arcee's MergeKit: A Toolkit for Merging Large Language Models](https://mlsanity.com/paper/2403.13257/) (2024)
- [CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages](https://mlsanity.com/paper/2309.09400/) (2023)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks](https://mlsanity.com/paper/2312.06795/) (2023)
- [What Language Model to Train if You Have One Million GPU Hours?](https://mlsanity.com/paper/2210.15424/) (2022)
- [Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages](https://mlsanity.com/paper/2310.04799/) (2023)
- [Latxa: An Open Language Model and Evaluation Suite for Basque](https://mlsanity.com/paper/2403.20266/) (2024)
- [A Comparative Study of Open-Source Large Language Models, GPT-4 and Claude 2: Multiple-Choice Test Taking in Nephrology](https://mlsanity.com/paper/2308.04709/) (2023)
- [BertaQA: How Much Do Language Models Know About Local Culture?](https://mlsanity.com/paper/2406.07302/) (2024)
- [Unlocking the Potential of Model Merging for Low-Resource Languages](https://mlsanity.com/paper/2407.03994/) (2024)
- [Better Alignment with Instruction Back-and-Forth Translation](https://mlsanity.com/paper/2408.04614/) (2024)
- [Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basque](https://mlsanity.com/paper/2506.07597/) (2025)
- [Adapting Language-Specific LLMs to a Reasoning Model in One Day via Model Merging -- An Open Recipe](https://mlsanity.com/paper/2502.09056/) (2025)
- [IberBench: LLM Evaluation on Iberian Languages](https://mlsanity.com/paper/2504.16921/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) remain heavily centered on English, with limited performance in low-resource languages. Existing adaptation approaches, such as continual pre-training, demand significant computational resources. In the case of instructed models, high-quality instruction data is also required, both of which are often inaccessible for low-resource language communities. Under these constraints, model merging offers a lightweight alternative, but its potential in low-resource contexts has not been systematically explored. In this work, we explore whether it is possible to transfer language knowledge to an instruction-tuned LLM by merging it with a language-specific base model, thereby eliminating the need of language-specific instructions and repeated fine-tuning processes whenever stronger instructed variants become available. Through experiments covering four Iberian languages (Basque, Catalan, Galician, and Spanish) and two model families, we show that merging enables effective instruction following behavior in new languages and even supports multilingual capability through the combination of multiple language-specific models. Our results indicate that model merging is a viable and efficient alternative to traditional adaptation methods for low-resource languages, achieving competitive performance while greatly reducing computational cost.