mlsanityData to 14 Oct 2026

Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization

arXiv 2311.09344 ↗First posted 15 Nov 2023cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

Parameter-efficient fine-tuning (PEFT) using labeled task data can significantly improve the performance of large language models (LLMs) on the downstream task. However, there are 7000 languages in the world and many of these languages lack labeled data for real-world language generation tasks. In this paper, we propose to improve zero-shot cross-lingual transfer by composing language or task specialized parameters. Our method composes language and task PEFT modules via element-wise arithmetic operations to leverage unlabeled data and English labeled data. We extend our approach to cases where labeled data from more languages is available and propose to arithmetically compose PEFT modules trained on languages related to the target. Empirical results on summarization demonstrate that our method is an effective strategy that obtains consistent gains using minimal training of PEFT modules.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 41 of its 57 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

41 of 57 references.

References · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
29 more
  1. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  2. 2015Teaching Machines to Read and Comprehendcited by 1,049
  3. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  4. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  5. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  6. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  7. 2017Learning multiple visual domains with residual adapterscited by 557
  8. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  9. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  10. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  11. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  12. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  13. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  14. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 294
  15. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
  16. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 242
  17. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 222
  18. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  19. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 156
  20. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  21. 2020MLSUM: The Multilingual Summarization Corpuscited by 121
  22. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  23. 2020UDapter: Language Adaptation for Truly Universal Dependency Parsingcited by 77
  24. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  25. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  26. 2020Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Wellcited by 57
  27. 2022Multi-Head Adapter Routing for Cross-Task Generalizationcited by 33
  28. 2020Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translationcited by 28
  29. 2020Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfercited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 7; 2026: 8.

28 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  2. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  3. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  4. 2024LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Taskscited by 39
  5. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 29
  6. 2024Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Mergingcited by 19
  7. 2024Cross-Lingual Auto Evaluation for Assessing Multilingual LLMscited by 18
  8. 2024Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfercited by 13
  9. 2024XL-HeadTags: Leveraging Multimodal Retrieval Augmentation for the Multilingual Generation of News Headlines and Tagscited by 9
  10. 2024DPPA: Pruning Method for Large Language Model to Model Mergingcited by 7
  11. 2025Multi-Task Vehicle Routing Solver via Mixture of Specialized Experts under State-Decomposable MDPcited by 7
  12. 2024No Train but Gain: Language Arithmetic for training-free Language Adapters enhancementcited by 5
16 more
  1. 2025Investigating Task Arithmetic for Zero-Shot Information Retrievalcited by 4
  2. 2025BADTV: Unveiling Backdoor Threats in Third-Party Task Vectorscited by 3
  3. 2026Layer-wise Swapping for Generalizable Multilingual Safetycited by 2
  4. 2026Bus-Conditioned Zero-Shot Trajectory Generation via Task Arithmeticcited by 2
  5. 2024AT-MoE: Adaptive Task-planning Mixture of Experts via LoRA Approachcited by 1
  6. 2025DivMerge: A divergence-based model merging method for multi-taskingcited by 1
  7. 2026Typologically Informed Parameter Aggregationcited by 1
  8. 2024Inducing Generalization across Languages and Tasks using Featurized Low-Rank Mixturescited by 0
  9. 2024Language-Independent Representations Improve Zero-Shot Summarizationcited by 0
  10. 2025Reverse Probing: Evaluating Knowledge Transfer via Finetuned Task Embeddings for Coreference Resolutioncited by 0
  11. 2025Position: Pause Recycling LoRAs and Prioritize Mechanisms to Uncover Limits and Effectivenesscited by 0
  12. 2026Task Arithmetic with Support Languages for Low-Resource ASRcited by 0
  13. 2026On the Limits of Model Merging for Multilinguality in Pre-Trainingcited by 0
  14. 2026Domain Arithmetic: One-Shot VLA Adaptation under Environmental Shiftscited by 0
  15. 2026One Domain, Many Tongues: Composing Domain and Language LoRAs for Cross-Lingual Remote-Sensing MLLMs without Paired Datacited by 0
  16. 2026Language Distances are Practical for Equitable Cross-Lingual Transfercited by 0