# Improving Training Efficiency and Reducing Maintenance Costs via Language Specific Model Merging

arXiv: https://arxiv.org/abs/2601.16127 · HTML: https://mlsanity.com/paper/2601.16127/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 20 references that are in the map.
- First posted on arXiv 22 Jan 2026.

## Builds on (most cited first, 10 of 10)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization](https://mlsanity.com/paper/2010.03093/) (2020)
- [Model merging with SVD to tie the Knots](https://mlsanity.com/paper/2410.19735/) (2024)
- [AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Merging](https://mlsanity.com/paper/2402.18913/) (2024)
- [MultiFiT: Efficient Multi-lingual Language Model Fine-tuning](https://mlsanity.com/paper/1909.04761/) (2019)
- [Improving Multilingual Instruction Finetuning via Linguistically Natural and Diverse Datasets](https://mlsanity.com/paper/2407.01853/) (2024)

## Built on by (most cited first, 2 of 2)

- [One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Merging](https://mlsanity.com/paper/2604.02881/) (2026)
- [From Parameters to Behaviors: A Survey of Model Fusion for Large Language Models](https://mlsanity.com/paper/2609.19553/) (2026)

## Abstract (arXiv)

Fine-tuning a task-specific multilingual large language model (LLM) involves training the model on a multilingual dataset with examples in all the required languages. Updating one or more supported languages with additional data or adding support for a new language involves retraining the model, which can be computationally inefficient and creates a severe maintenance bottleneck. Recent research on merging multilingual multitask models has shown promise in terms of improved quality, but its computational and maintenance efficiency remains unstudied. In this work, we provide the first focused analysis of this merging strategy from an efficiency perspective, evaluating it across three independent tasks. We demonstrate significant efficiency gains while maintaining parity in terms of quality: this merging approach reduces the initial training time by up to 50%. We also demonstrate that updating an individual language and re-merging as part of model maintenance reduces training costs by more than 60%, compared to re-training the full multilingual model. We show this on both public and proprietary industry datasets confirming that the approach works well for industrial use cases in addition to academic settings already studied in previous work.