mlsanityData to 14 Oct 2026

DeltaMerge-LowRes: Composing Language and Task Deltas for Low-Resource Adaptation

arXiv 2607.13967 ↗First posted 15 Jul 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Adapting a multilingual encoder to a new language and a new task with only a few hundred gold examples is a common low-resource NLP setting, yet the two axes are usually fused via an expensive language--task fine-tuning run. We ask whether they can instead be trained separately and recombined in weight space. DeltaMergeLowRes learns a language delta DeltaL from unlabeled monolingual text and a task delta DeltaT from labeled English data, then composes them at inference under one of four rules: additive, activation-guided, sparsity-aware, and a novel cross-axis TIES. The new rule adapts the TIES-Merging steps of trimming, sign election, and merging to the language and task axes rather than to two task axes. Holding (DeltaL,DeltaT) fixed across rules on four task families and four African languages (158 evaluated cells, 10,000-sample paired bootstrap per cell), we find: (i) cross-axis TIES wins summarisation on 3/4 languages by +4 to +7 chrF (chrF 18.59 vs. 13.80 task-only); (ii) it improves QA F1 by +2.32 and EM by +2.91; and (iii) sparsity-aware merging cuts classification ECE by 36% at parity macro-F1. The composition rule materially changes what the merged model preserves, suppresses, and calibrates. We release all JSON traces and a claim ledger.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 29 references that are in the map. First posted on arXiv 15 Jul 2026.

Builds on

21 of 29 references.

References · 21

  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  8. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  9. 2021Robust fine-tuning of zero-shot modelscited by 797
  10. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  11. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  12. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
9 more
  1. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  2. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  3. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  4. 2021Training Neural Networks with Fixed Sparse Maskscited by 214
  5. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  6. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  7. 2023MasakhaNEWS: News Topic Classification for African languagescited by 31
  8. 2023AfriQA: Cross-lingual Open-Retrieval Question Answering for African Languagescited by 23
  9. 2022Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfercited by 22

Built on by

Not cited yet.