mlsanityData to 7 Oct 2026

Are All Languages Created Equal in Multilingual BERT?

arXiv 2005.09093 ↗First posted 18 May 2020cs.CLcited by 206

Not on the map. Why?

Abstract (arXiv)

Multilingual BERT (mBERT) trained on 104 languages has shown surprisingly good cross-lingual performance on several NLP tasks, even without explicit cross-lingual signals. However, these evaluations have focused on cross-lingual transfer with high-resource languages, covering only a third of the languages covered by mBERT. We explore how mBERT performs on a much wider set of languages, focusing on the quality of representation for low-resource languages, measured by within-language performance. We consider three tasks: Named Entity Recognition (99 languages), Part-of-speech Tagging, and Dependency Parsing (54 languages each). mBERT does better than or comparable to baselines on high resource languages but does much worse for low resource languages. Furthermore, monolingual BERT models for these languages do even worse. Paired with similar languages, the performance gap between monolingual BERT and mBERT can be narrowed. We find that better models for low resource languages require more efficient pretraining techniques or more data.

Key facts. Cited by 206 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 33; 2022: 42; 2023: 41; 2024: 40; 2025: 23; 2026: 27.

206 papers.

Sort:

By area:

  1. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  2. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  3. 2021MuRIL: Multilingual Representations for Indian Languagescited by 177
  4. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  5. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  6. 2021BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Banglacited by 123
  7. 2022The Geometry of Multilingual Language Model Representationscited by 94
  8. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 94
  9. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  10. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 92
  11. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  12. 2021Mitigating Language-Dependent Ethnic Bias in BERTcited by 87
38 more
  1. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  2. 2023Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpacacited by 60
  3. 2022Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP Worldcited by 55
  4. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  5. 2022Language Modelling with Pixelscited by 53
  6. 2021AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languagescited by 50
  7. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  8. 2024Probing the Emergence of Cross-lingual Alignment during LLM Trainingcited by 38
  9. 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 36
  10. 2021Assessing Multilingual Fairness in Pre-trained Multimodal Representationscited by 35
  11. 2023Lost in Translation: Large Language Models in Non-English Content Analysiscited by 34
  12. 2022Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Modelscited by 32
  13. 2023When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languagescited by 32
  14. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  15. 2022Enhancing Cross-lingual Transfer by Manifold Mixupcited by 30
  16. 2022A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bankcited by 29
  17. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 27
  18. 2022Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferabilitycited by 26
  19. 2023Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmarkcited by 26
  20. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignmentcited by 26
  21. 2022Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languagescited by 25
  22. 2021Predicting the Performance of Multilingual NLP Modelscited by 24
  23. 2022GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Costcited by 24
  24. 2022Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Trainingcited by 24
  25. 2023mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinationscited by 24
  26. 2021Operationalizing a National Digital Library: The Case for a Norwegian Transformer Modelcited by 23
  27. 2021Specializing Multilingual Language Models: An Empirical Studycited by 21
  28. 2022Model and Data Transfer for Cross-Lingual Sequence Labelling in Zero-Resource Settingscited by 21
  29. 2024IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generatorscited by 21
  30. 2022Multi Task Learning For Zero Shot Performance Prediction of Multilingual Modelscited by 20
  31. 2022Label Semantic Aware Pre-training for Few-shot Text Classificationcited by 18
  32. 2022On the Calibration of Massively Multilingual Language Modelscited by 18
  33. 2023Exploring the Relationship between Alignment and Cross-lingual Transfer in Multilingual Transformerscited by 18
  34. 2022Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languagescited by 17
  35. 2021Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systemscited by 16
  36. 2021Everything Is All It Takes: A Multipronged Strategy for Zero-Shot Cross-Lingual Information Extractioncited by 16
  37. 2023Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languagescited by 16
  38. 2024MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routingcited by 16

50 most cited of 206 All 206 (JSON)