mlsanityData to 7 Oct 2026

MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer

arXiv 2005.00052 ↗First posted 30 Apr 2020cs.CLcited by 443

Not on the map. Why?

Abstract (arXiv)

The main goal behind state-of-the-art pre-trained multilingual models such as multilingual BERT and XLM-R is enabling and bootstrapping NLP applications in low-resource languages through zero-shot or few-shot cross-lingual transfer. However, due to limited model capacity, their transfer performance is the weakest exactly on such low-resource languages and languages unseen during pre-training. We propose MAD-X, an adapter-based framework that enables high portability and parameter-efficient transfer to arbitrary tasks and languages by learning modular language and task representations. In addition, we introduce a novel invertible adapter architecture and a strong baseline method for adapting a pre-trained multilingual model to a new language. MAD-X outperforms the state of the art in cross-lingual transfer across a representative set of typologically diverse languages on named entity recognition and causal commonsense reasoning, and achieves competitive results on question answering. Our code and adapters are available at AdapterHub.ml

Key facts. Cited by 443 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 60; 2022: 85; 2023: 104; 2024: 101; 2025: 55; 2026: 38.

443 papers.

Sort:

By area:

  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  3. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  4. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  5. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  6. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  7. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  8. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  9. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  10. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  11. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 167
  12. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
38 more
  1. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  2. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  3. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  4. 2024How do Large Language Models Handle Multilingualism?cited by 148
  5. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  6. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  7. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145
  8. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  9. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  10. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  11. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  12. 2021Sustainable Modular Debiasing of Language Modelscited by 98
  13. 2021Language Models are Few-shot Multilingual Learnerscited by 92
  14. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  15. 2021MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfercited by 89
  16. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  17. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  18. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  19. 2023Modular Deep Learningcited by 82
  20. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  21. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  22. 2024Multilingual Instruction Tuning With Just a Pinch of Multilingualitycited by 71
  23. 2021Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processingcited by 68
  24. 2023Structure-informed Language Models Are Protein Designerscited by 68
  25. 2022NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysiscited by 64
  26. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 62
  27. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  28. 2021PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domainscited by 61
  29. 2021Multi-Task Learning in Natural Language Processing: An Overviewcited by 61
  30. 2021Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmarkcited by 60
  31. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  32. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  33. 2024MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Modelscited by 54
  34. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 53
  35. 2022Language Modelling with Pixelscited by 53
  36. 2023Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learningcited by 53
  37. 2021AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languagescited by 50
  38. 2021A Primer on Pretrained Multilingual Language Modelscited by 49

50 most cited of 443 All 443 (JSON)