mlsanityData to 7 Oct 2026

How to Adapt Your Pretrained Multilingual Model to 1600 Languages

arXiv 2106.02124 ↗First posted 3 Jun 2021cs.CLcited by 54

Not on the map. Why?

Abstract (arXiv)

Pretrained multilingual models (PMMs) enable zero-shot learning via cross-lingual transfer, performing best for languages seen during pretraining. While methods exist to improve performance for unseen languages, they have almost exclusively been evaluated using amounts of raw text only available for a small fraction of the world's languages. In this paper, we evaluate the performance of existing methods to adapt PMMs to new languages using a resource available for over 1600 languages: the New Testament. This is challenging for two reasons: (1) the small corpus size, and (2) the narrow domain. While performance drops for all approaches, we surprisingly still see gains of up to 17.69% accuracy for part-of-speech tagging and 6.29 F1 for NER on average over all languages as compared to XLM-R. Another unexpected finding is that continued pretraining, the simplest approach, performs best. Finally, we perform a case study to disentangle the effects of domain and size and to shed light on the influence of the finetuning source language.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 35 of its 47 references that are in the map. First posted on arXiv 3 Jun 2021.

Builds on

35 of 47 references.

References · 35

  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  8. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  10. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  11. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  12. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
23 more
  1. 2019How multilingual is Multilingual BERT?cited by 642
  2. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  3. 2017Learning multiple visual domains with residual adapterscited by 553
  4. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  5. 2017Word Translation Without Parallel Datacited by 480
  6. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  7. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  8. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  9. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  10. 2018Learning Word Vectors for 157 Languagescited by 279
  11. 2019Measuring Bias in Contextualized Word Representationscited by 275
  12. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  13. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 269
  14. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  15. 2017A Survey Of Cross-lingual Word Embedding Modelscited by 142
  16. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  17. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  18. 2018Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterioncited by 69
  19. 2018Neural Cross-Lingual Named Entity Recognition with Minimal Resourcescited by 44
  20. 2016Massively Multilingual Word Embeddingscited by 26
  21. 2020Multi-Stage Pre-training for Low-Resource Domain Adaptationcited by 24
  22. 2019Zero-Resource Cross-Lingual Named Entity Recognitioncited by 18
  23. 2020Parsing with Multilingual BERT, a Small Corpus, and a Small Treebankcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 15; 2023: 12; 2024: 15; 2025: 8; 2026: 3.

54 papers.

Sort:

  1. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  2. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  3. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  4. 2022Language Modelling with Pixelscited by 53
  5. 2022Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptationcited by 44
  6. 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 36
  7. 2023Efficiently Adapting Pretrained Language Models To New Languagescited by 28
  8. 2024MaLA-500: Massive Language Adaptation of Large Language Modelscited by 28
  9. 2022Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferabilitycited by 26
  10. 2022Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation?cited by 24
  11. 2022GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Costcited by 24
  12. 2024Exploring Design Choices for Building Language-Specific LLMscited by 22
38 more
  1. 2023InstructAlign: High-and-Low Resource Language Alignment via Continual Crosslingual Instruction Tuningcited by 21
  2. 2023Sabi'a: Portuguese Large Language Modelscited by 19
  3. 2023Adapting Multilingual Speech Representation Model for a New, Underresourced Language through Multilingual Fine-tuning and Continued Pretrainingcited by 18
  4. 2023Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languagescited by 16
  5. 2022BERTifying Sinhala -- A Comprehensive Analysis of Pre-trained Language Models for Sinhala Text Classificationcited by 15
  6. 2022A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Modelscited by 12
  7. 2022MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learningcited by 12
  8. 2024SambaLingo: Teaching Large Language Models New Languagescited by 11
  9. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 11
  10. 2023Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretrainingcited by 10
  11. 2024Targeted Multilingual Adaptation for Low-resource Language Familiescited by 9
  12. 2023A Survey on Arabic Named Entity Recognition: Past, Recent Advances, and Future Trendscited by 8
  13. 2022MulZDG: Multilingual Code-Switching Framework for Zero-shot Dialogue Generationcited by 7
  14. 2022Adapting BigScience Multilingual Model to Unseen Languagescited by 6
  15. 2024The Impact of Language Adapters in Cross-Lingual Transfer for NLUcited by 6
  16. 2025ConLID: Supervised Contrastive Learning for Low-Resource Language Identificationcited by 5
  17. 2022Graph-Based Multilingual Label Propagation for Low-Resource Part-of-Speech Taggingcited by 4
  18. 2023Meeting the Needs of Low-Resource Languages: The Value of Automatic Alignments via Pretrained Modelscited by 4
  19. 2023MC2: Towards Transparent and Culturally-Aware NLP for Minority Languages in Chinacited by 4
  20. 2024Do Language Models Care About Text Quality? Evaluating Web-Crawled Corpora Across 11 Languagescited by 4
  21. 2024Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretrainingcited by 4
  22. 2024Comparing LLM prompting with Cross-lingual transfer performance on Indigenous and Low-resource Brazilian Languagescited by 4
  23. 2024UniBridge: A Unified Approach to Cross-Lingual Transfer Learning for Low-Resource Languagescited by 4
  24. 2022Extending the Subwording Model of Multilingual Pretrained Models for New Languagescited by 3
  25. 2023Extending Multilingual Machine Translation through Imitation Learningcited by 3
  26. 2024Tuning LLMs with Contrastive Alignment Instructions for Machine Translation in Unseen, Low-resource Languagescited by 3
  27. 2025Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languagescited by 3
  28. 2025Humans overrely on overconfident language models, across languagescited by 3
  29. 2025Model-Based Ranking of Source Languages for Zero-Shot Cross-Lingual Transfercited by 3
  30. 2022Dim Wihl Gat Tun: The Case for Linguistic Expertise in NLP for Underdocumented Languagescited by 2
  31. 2022A Comparative Study of Pre-trained Encoders for Low-Resource Named Entity Recognitioncited by 1
  32. 2024HumVI: A Multilingual Dataset for Detecting Violent Incidents Impacting Humanitarian Aidcited by 1
  33. 2025Sparse Subnetwork Enhancement for Underrepresented Languages in Large Language Modelscited by 1
  34. 2023Syntactic Inductive Bias in Transformer Language Models: Especially Helpful for Low-Resource Languages?cited by 0
  35. 2024Large language model for Bible sentiment analysis: Sermon on the Mountcited by 0
  36. 2024Evaluation Methodology for Large Language Models for Multilingual Document Question and Answercited by 0
  37. 2024Efficacy of ByT5 in Multilingual Translation of Biblical Texts for Underrepresented Languagescited by 0
  38. 2025Multilinguality Does not Make Sense: Investigating Factors Behind Zero-Shot Transfer in Sense-Aware Taskscited by 0

50 most cited of 54 All 54 (JSON)