# No Train but Gain: Language Arithmetic for training-free Language Adapters enhancement

arXiv: https://arxiv.org/abs/2404.15737 · HTML: https://mlsanity.com/paper/2404.15737/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 51 references that are in the map.
- First posted on arXiv 24 Apr 2024.

## Builds on (most cited first, 30 of 36)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition](https://mlsanity.com/paper/2307.13269/) (2023)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [ZipIt! Merging Models from Different Tasks without Training](https://mlsanity.com/paper/2305.03053/) (2023)
- [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://mlsanity.com/paper/2205.06266/) (2022)
- [Composable Sparse Fine-Tuning for Cross-Lingual Transfer](https://mlsanity.com/paper/2110.07560/) (2021)
- [Survey of Low-Resource Machine Translation](https://mlsanity.com/paper/2109.00486/) (2021)
- [AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models](https://mlsanity.com/paper/2302.07027/) (2023)
- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [UDapter: Language Adaptation for Truly Universal Dependency Parsing](https://mlsanity.com/paper/2004.14327/) (2020)
- [What the [MASK]? Making Sense of Language-Specific BERT Models](https://mlsanity.com/paper/2003.02912/) (2020)
- [Learning Language Representations for Typology Prediction](https://mlsanity.com/paper/1707.09569/) (2017)

## Built on by (most cited first, 5 of 5)

- [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024)
- [Parameter Competition Balancing for Model Merging](https://mlsanity.com/paper/2410.02396/) (2024)
- [Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfer](https://mlsanity.com/paper/2408.01119/) (2024)
- [Soft Language Prompts for Language Transfer](https://mlsanity.com/paper/2407.02317/) (2024)
- [Typologically Informed Parameter Aggregation](https://mlsanity.com/paper/2601.16629/) (2026)

## Abstract (arXiv)

Modular deep learning is the state-of-the-art solution for lifting the curse of multilinguality, preventing the impact of negative interference and enabling cross-lingual performance in Multilingual Pre-trained Language Models. However, a trade-off of this approach is the reduction in positive transfer learning from closely related languages. In response, we introduce a novel method called language arithmetic, which enables training-free post-processing to address this limitation. Extending the task arithmetic framework, we apply learning via addition to the language adapters, transitioning the framework from a multi-task to a multilingual setup. The effectiveness of the proposed solution is demonstrated on three downstream tasks in a MAD-X-based set of cross-lingual schemes, acting as a post-processing procedure. Language arithmetic consistently improves the baselines with significant gains, especially in the most challenging case of zero-shot application. Our code and models are available at https://github.com/mklimasz/language-arithmetic .