# DeltaMerge-LowRes: Composing Language and Task Deltas for Low-Resource Adaptation

arXiv: https://arxiv.org/abs/2607.13967 · HTML: https://mlsanity.com/paper/2607.13967/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 15 Jul 2026.

## Builds on (most cited first, 21 of 21)

- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Dataless Knowledge Fusion by Merging Weights of Language Models](https://mlsanity.com/paper/2212.09849/) (2022)
- [Parameter-Efficient Transfer Learning with Diff Pruning](https://mlsanity.com/paper/2012.07463/) (2020)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [Training Neural Networks with Fixed Sparse Masks](https://mlsanity.com/paper/2111.09839/) (2021)
- [Composable Sparse Fine-Tuning for Cross-Lingual Transfer](https://mlsanity.com/paper/2110.07560/) (2021)
- [A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages](https://mlsanity.com/paper/2006.06202/) (2020)
- [MasakhaNEWS: News Topic Classification for African languages](https://mlsanity.com/paper/2304.09972/) (2023)
- [AfriQA: Cross-lingual Open-Retrieval Question Answering for African Languages](https://mlsanity.com/paper/2305.06897/) (2023)
- [Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfer](https://mlsanity.com/paper/2205.12148/) (2022)

## Abstract (arXiv)

Adapting a multilingual encoder to a new language and a new task with only a few hundred gold examples is a common low-resource NLP setting, yet the two axes are usually fused via an expensive language--task fine-tuning run. We ask whether they can instead be trained separately and recombined in weight space. DeltaMergeLowRes learns a language delta DeltaL from unlabeled monolingual text and a task delta DeltaT from labeled English data, then composes them at inference under one of four rules: additive, activation-guided, sparsity-aware, and a novel cross-axis TIES. The new rule adapts the TIES-Merging steps of trimming, sign election, and merging to the language and task axes rather than to two task axes. Holding (DeltaL,DeltaT) fixed across rules on four task families and four African languages (158 evaluated cells, 10,000-sample paired bootstrap per cell), we find: (i) cross-axis TIES wins summarisation on 3/4 languages by +4 to +7 chrF (chrF 18.59 vs. 13.80 task-only); (ii) it improves QA F1 by +2.32 and EM by +2.91; and (iii) sparsity-aware merging cuts classification ECE by 36% at parity macro-F1. The composition rule materially changes what the merged model preserves, suppresses, and calibrates. We release all JSON traces and a claim ledger.