mlsanityData to 7 Oct 2026

How multilingual is Multilingual BERT?

arXiv 1906.01502 ↗First posted 4 Jun 2019cs.CL cs.AI cs.LGcited by 642

Not on the map. Why?

Abstract (arXiv)

In this paper, we show that Multilingual BERT (M-BERT), released by Devlin et al. (2018) as a single language model pre-trained from monolingual corpora in 104 languages, is surprisingly good at zero-shot cross-lingual model transfer, in which task-specific annotations in one language are used to fine-tune the model for evaluation in another language. To understand why, we present a large number of probing experiments, showing that transfer is possible even to languages in different scripts, that transfer works best between typologically similar languages, that monolingual corpora can train models for code-switching, and that the model can find translation pairs. From these results, we can conclude that M-BERT does create multilingual representations, but that these representations exhibit systematic deficiencies affecting certain language pairs.

Key facts. Cited by 642 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 136; 2022: 128; 2023: 111; 2024: 111; 2025: 73; 2026: 83.

642 papers.

Sort:

By area:

  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  2. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  3. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  4. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  5. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  6. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  7. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  8. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  9. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  10. 2022Translation between Molecules and Natural Languagecited by 173
  11. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  12. 2023Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Modelscited by 144
38 more
  1. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 142
  2. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  3. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  4. 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 124
  5. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  6. 2021XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyondcited by 114
  7. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmarkcited by 114
  8. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  9. 2022The Geometry of Multilingual Language Model Representationscited by 94
  10. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 94
  11. 2021Learning Compact Metrics for MTcited by 93
  12. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  13. 2022Character-Aware Models Improve Visual Text Renderingcited by 90
  14. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  15. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  16. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  17. 2023Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptationcited by 79
  18. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  19. 2024Multilingual Instruction Tuning With Just a Pinch of Multilingualitycited by 71
  20. 2021It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoningcited by 69
  21. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 65
  22. 2021First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERTcited by 64
  23. 2022A Survey on Aspect-Based Sentiment Analysis: Tasks, Methods, and Challengescited by 61
  24. 2023A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologiescited by 59
  25. 2021Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completioncited by 57
  26. 2022Transferability in Deep Learning: A Surveycited by 57
  27. 2021Are Multilingual Models Effective in Code-Switching?cited by 56
  28. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  29. 2024ALLaM: Large Language Models for Arabic and Englishcited by 55
  30. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  31. 2022Language Modelling with Pixelscited by 53
  32. 2022Multi-VALUE: A Framework for Cross-Dialectal English NLPcited by 51
  33. 2021AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languagescited by 50
  34. 2023Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Reviewcited by 50
  35. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  36. 2021Discrete and Soft Prompting for Multilingual Modelscited by 49
  37. 2022L3Cube-MahaCorpus and MahaBERT: Marathi Monolingual Corpus, Marathi BERT Language Models, and Resourcescited by 49
  38. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 48

50 most cited of 642 All 642 (JSON)