mlsanityData to 14 Oct 2026

Extending Multilingual BERT to Low-Resource Languages

arXiv 2004.13640 ↗First posted 28 Apr 2020cs.CLcited by 88

Not on the map. Why?

Abstract (arXiv)

Multilingual BERT (M-BERT) has been a huge success in both supervised and zero-shot cross-lingual transfer learning. However, this success has focused only on the top 104 languages in Wikipedia that it was trained on. In this paper, we propose a simple but effective approach to extend M-BERT (E-BERT) so that it can benefit any new language, and show that our approach benefits languages that are already in M-BERT as well. We perform an extensive set of experiments with Named Entity Recognition (NER) on 27 languages, only 16 of which are in M-BERT, and show an average increase of about 6% F1 on languages that are already in M-BERT and 23% F1 increase on new languages.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). First posted on arXiv 28 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 15; 2022: 23; 2023: 15; 2024: 19; 2025: 9; 2026: 7.

88 papers.

Sort:

By area:

  1. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 232
  2. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  3. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 132
  4. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  5. 2022A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translationcited by 98
  6. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  7. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 84
  8. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  9. 2022Language Modelling with Pixelscited by 55
  10. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  11. 2021AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languagescited by 50
  12. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
38 more
  1. 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languagescited by 48
  2. 2021Revisiting the Primacy of English in Zero-shot Cross-lingual Transfercited by 47
  3. 2022Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptationcited by 44
  4. 2023Lost in Translation: Large Language Models in Non-English Content Analysiscited by 35
  5. 2021Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translationcited by 29
  6. 2022Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifoldcited by 29
  7. 2023Dissociating language and thought in large language modelscited by 29
  8. 2024MaLA-500: Massive Language Adaptation of Large Language Modelscited by 28
  9. 2022Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferabilitycited by 26
  10. 2021IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initializationcited by 25
  11. 2022Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Trainingcited by 24
  12. 2021Specializing Multilingual Language Models: An Empirical Studycited by 21
  13. 2021Exploiting Language Relatedness for Low Web-Resource Language Model Adaptation: An Indic Languages Studycited by 19
  14. 2024An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inferencecited by 18
  15. 2023Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languagescited by 16
  16. 2022USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translationcited by 15
  17. 2022Intriguing Properties of Compression on Multilingual Modelscited by 15
  18. 2023Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillationcited by 15
  19. 2023Translation and Fusion Improves Zero-shot Cross-lingual Information Extractioncited by 15
  20. 2022Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltesecited by 14
  21. 2022A Warm Start and a Clean Crawled Corpus -- A Recipe for Good Language Modelscited by 13
  22. 2021Subword Mapping and Anchoring across Languagescited by 12
  23. 2022A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Modelscited by 12
  24. 2022MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learningcited by 12
  25. 2024GlossLM: A Massively Multilingual Corpus and Pretrained Model for Interlinear Glossed Textcited by 12
  26. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 12
  27. 2023Soft Prompt Decoding for Multilingual Dense Retrievalcited by 10
  28. 2023Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AIcited by 9
  29. 2024Targeted Multilingual Adaptation for Low-resource Language Familiescited by 9
  30. 2024Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?cited by 9
  31. 2024Pre-training Differentially Private Models with Limited Public Datacited by 8
  32. 2024Zero-shot Sentiment Analysis in Low-Resource Languages Using a Multilingual Sentiment Lexiconcited by 7
  33. 2021Do Multilingual Neural Machine Translation Models Contain Language Pair Specific Attention Heads?cited by 6
  34. 2022Adapting BigScience Multilingual Model to Unseen Languagescited by 6
  35. 2022Improving Low-Resource Speech Recognition with Pretrained Speech Models: Continued Pretraining vs. Semi-Supervised Trainingcited by 6
  36. 2024NSINA: A News Corpus for Sinhalacited by 6
  37. 2024LogogramNLP: Comparing Visual and Textual Representations of Ancient Logographic Writing Systems for NLPcited by 6
  38. 2025Overcoming Vocabulary Constraints with Pixel-level Fallbackcited by 6

50 most cited of 88 All 88 (JSON)