mlsanityData to 14 Oct 2026

When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models

arXiv 2010.12858 ↗First posted 24 Oct 2020cs.CLcited by 131

Not on the map. Why?

Abstract (arXiv)

Transfer learning based on pretraining language models on a large amount of raw data has become a new norm to reach state-of-the-art performance in NLP. Still, it remains unclear how this approach should be applied for unseen languages that are not covered by any available large-scale multilingual language model and for which only a small amount of raw data is generally available. In this work, by comparing multilingual and monolingual models, we show that such models behave in multiple ways on unseen languages. Some languages greatly benefit from transfer learning and behave similarly to closely related high resource languages whereas others apparently do not. Focusing on the latter, we show that this failure to transfer is largely related to the impact of the script used to write such languages. Transliterating those languages improves very significantly the ability of large-scale multilingual language models on downstream tasks.

Key facts. Cited by 131 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 14; 2022: 28; 2023: 24; 2024: 33; 2025: 20; 2026: 12.

131 papers.

Sort:

By area:

  1. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  2. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  3. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  4. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  5. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  6. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  7. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 120
  8. 2021Survey of Low-Resource Machine Translationcited by 92
  9. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  10. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 86
  11. 2025Do Multilingual LLMs Think In English?cited by 78
  12. 2022NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysiscited by 64
38 more
  1. 2022Language Modelling with Pixelscited by 55
  2. 2022Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP Worldcited by 55
  3. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  4. 2021AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languagescited by 50
  5. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  6. 2022Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptationcited by 44
  7. 2021From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understandingcited by 43
  8. 2023Lost in Translation: Large Language Models in Non-English Content Analysiscited by 35
  9. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  10. 2024RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanizationcited by 29
  11. 2024MaLA-500: Massive Language Adaptation of Large Language Modelscited by 28
  12. 2022Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferabilitycited by 26
  13. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignmentcited by 26
  14. 2022Phylogeny-Inspired Adaptation of Multilingual Models to New Languagescited by 24
  15. 2022GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Costcited by 24
  16. 2022Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Trainingcited by 24
  17. 2021Dataset Geography: Mapping Language Data to Language Userscited by 22
  18. 2024How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?cited by 22
  19. 2021Specializing Multilingual Language Models: An Empirical Studycited by 21
  20. 2023Romanization-based Large-scale Adaptation of Multilingual Language Modelscited by 21
  21. 2023InstructAlign: High-and-Low Resource Language Alignment via Continual Crosslingual Instruction Tuningcited by 21
  22. 2023An Efficient Multilingual Language Model Compression through Vocabulary Trimmingcited by 20
  23. 2021Improving Zero-shot Cross-lingual Transfer between Closely Related Languages by injecting Character-level Noisecited by 19
  24. 2022Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Userscited by 19
  25. 2024An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inferencecited by 18
  26. 2023Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languagescited by 16
  27. 2022Intriguing Properties of Compression on Multilingual Modelscited by 15
  28. 2024Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapterscited by 15
  29. 2021Alternative Input Signals Ease Transfer in Multilingual Machine Translationcited by 14
  30. 2022Does Transliteration Help Multilingual Language Modeling?cited by 14
  31. 2022Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltesecited by 14
  32. 2024Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignmentcited by 14
  33. 2024Multilingual Large Language Models and Curse of Multilingualitycited by 13
  34. 2021Subword Mapping and Anchoring across Languagescited by 12
  35. 2022MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learningcited by 12
  36. 2023Multilingual large language models leak human stereotypes across language boundariescited by 12
  37. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 12
  38. 2021Can Character-based Language Models Improve Downstream Task Performance in Low-Resource and Noisy Language Scenarios?cited by 11

50 most cited of 131 All 131 (JSON)