mlsanityData to 7 Oct 2026

Soft Language Prompts for Language Transfer

arXiv 2407.02317 ↗First posted 2 Jul 2024cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Cross-lingual knowledge transfer, especially between high- and low-resource languages, remains challenging in natural language processing (NLP). This study offers insights for improving cross-lingual NLP applications through the combination of parameter-efficient fine-tuning methods. We systematically explore strategies for enhancing cross-lingual transfer through the incorporation of language-specific and task-specific adapters and soft prompts. We present a detailed investigation of various combinations of these methods, exploring their efficiency across 16 languages, focusing on 10 mid- and low-resource languages. We further present to our knowledge the first use of soft prompts for language transfer, a technique we call soft language prompts. Our findings demonstrate that in contrast to claims of previous work, a combination of language and task adapters does not always work best; instead, combining a soft language prompt with a task adapter outperforms most configurations in many cases.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 42 of its 65 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

42 of 65 references.

References · 42

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  8. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  9. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  10. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  11. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  12. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
30 more
  1. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  2. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  3. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  4. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  5. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  6. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  7. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  8. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  9. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  10. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  11. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
  12. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  13. 2020UDapter: Language Adaptation for Truly Universal Dependency Parsingcited by 77
  14. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  15. 2024Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuningcited by 59
  16. 2023Multilingual Previously Fact-Checked Claim Retrievalcited by 32
  17. 2022Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Promptcited by 25
  18. 2023mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinationscited by 24
  19. 2024Discovering Low-rank Subspaces for Language-agnostic Multilingual Representationscited by 18
  20. 2024The Hidden Space of Transformer Language Adapterscited by 16
  21. 2024Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapterscited by 14
  22. 2023ZGUL: Zero-shot Generalization to Unseen Languages using Multi-source Ensembling of Language Adapterscited by 10
  23. 2024UQA: Corpus for Urdu Question Answeringcited by 8
  24. 2024A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpuscited by 8
  25. 2024Soft Prompt Tuning for Cross-Lingual Transfer: When Less is Morecited by 7
  26. 2024The Impact of Language Adapters in Cross-Lingual Transfer for NLUcited by 6
  27. 2024No Train but Gain: Language Arithmetic for training-free Language Adapters enhancementcited by 5
  28. 2020Reading Comprehension in Czech via Machine Translation and Cross-lingual Transfercited by 4
  29. 2023Prompt Learning to Mitigate Catastrophic Forgetting in Cross-lingual Transfer for Open-domain Dialogue Generationcited by 4
  30. 2023Multilingual and Multi-topical Benchmark of Fine-tuned Language models and Large Language Models for Check-Worthy Claim Detectioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2024GrEmLIn: A Repository of Green Baseline Embeddings for 87 Low-Resource Languages Injected with Multilingual Graph Knowledgecited by 7
  2. 2025Enhancing Cross-task Transfer of Large Language Models via Activation Steeringcited by 6
  3. 2025Sparse Subnetwork Enhancement for Underrepresented Languages in Large Language Modelscited by 1
  4. 2025Zero-Shot Cross-Lingual Transfer using Prefix-Based Adaptationcited by 0
  5. 2026Optimizing Soft Prompt Tuning via Structural Evolutioncited by 0