mlsanityData to 7 Oct 2026

The Impact of Language Adapters in Cross-Lingual Transfer for NLU

arXiv 2402.00149 ↗First posted 31 Jan 2024cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Modular deep learning has been proposed for the efficient adaption of pre-trained models to new tasks, domains and languages. In particular, combining language adapters with task adapters has shown potential where no supervised data exists for a language. In this paper, we explore the role of language adapters in zero-shot cross-lingual transfer for natural language understanding (NLU) benchmarks. We study the effect of including a target-language adapter in detailed ablation studies with two multilingual models and three multilingual datasets. Our results show that the effect of target-language adapters is highly inconsistent across tasks, languages and models. Retaining the source-language adapter instead often leads to an equivalent, and sometimes to a better, performance. Removing the language adapter after training has only a weak negative effect, indicating that the language adapters do not have a strong impact on the predictions.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 24 of its 40 references that are in the map. First posted on arXiv 31 Jan 2024.

Builds on

24 of 40 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  8. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  9. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  10. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  11. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  12. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
12 more
  1. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  2. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  3. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  4. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  5. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  6. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  7. 2020UDapter: Language Adaptation for Truly Universal Dependency Parsingcited by 77
  8. 2020UnNatural Language Inferencecited by 68
  9. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  10. 2023Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?cited by 39
  11. 2022Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferabilitycited by 26
  12. 2022Phylogeny-Inspired Adaptation of Multilingual Models to New Languagescited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 1.

6 papers.

  1. 2024The Hidden Space of Transformer Language Adapterscited by 16
  2. 2024Soft Language Prompts for Language Transfercited by 5
  3. 2024UniBridge: A Unified Approach to Cross-Lingual Transfer Learning for Low-Resource Languagescited by 4
  4. 2025Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languagescited by 3
  5. 2026Typologically Informed Parameter Aggregationcited by 1
  6. 2025How to Tune a Multilingual Encoder Model for Germanic Languages: A Study of PEFT, Full Fine-Tuning, and Language Adapterscited by 0