mlsanityData to 7 Oct 2026

Cross-Lingual Ability of Multilingual BERT: An Empirical Study

arXiv 1912.07840 ↗First posted 17 Dec 2019cs.CL cs.AI cs.LGcited by 182

Not on the map. Why?

Abstract (arXiv)

Recent work has exhibited the surprising cross-lingual abilities of multilingual BERT (M-BERT) -- surprising since it is trained without any cross-lingual objective and with no aligned data. In this work, we provide a comprehensive study of the contribution of different components in M-BERT to its cross-lingual ability. We study the impact of linguistic properties of the languages, the architecture of the model, and the learning objectives. The experimental study is done in the context of three typologically different languages -- Spanish, Hindi, and Russian -- and using two conceptually different NLP tasks, textual entailment and named entity recognition. Among our key conclusions is the fact that the lexical overlap between languages plays a negligible role in the cross-lingual success, while the depth of the network is an integral part of it. All our models and implementations can be found on our project page: http://cogcomp.org/page/publication_view/900 .

Key facts. Cited by 182 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 53; 2022: 39; 2023: 32; 2024: 25; 2025: 14; 2026: 19.

182 papers.

Sort:

By area:

  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  2. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  3. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  4. 2024How do Large Language Models Handle Multilingualism?cited by 148
  5. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  6. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 142
  7. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  8. 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 124
  9. 2021BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Banglacited by 123
  10. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 92
  11. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  12. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
38 more
  1. 2024Multilingual Instruction Tuning With Just a Pinch of Multilingualitycited by 71
  2. 2022A Survey on Aspect-Based Sentiment Analysis: Tasks, Methods, and Challengescited by 61
  3. 2023Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Biascited by 57
  4. 2024ALLaM: Large Language Models for Arabic and Englishcited by 55
  5. 2023Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Reviewcited by 50
  6. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  7. 2023EcomGPT: Instruction-tuning Large Language Models with Chain-of-Task Tasks for E-commercecited by 48
  8. 2021Revisiting the Primacy of English in Zero-shot Cross-lingual Transfercited by 47
  9. 2021MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairscited by 41
  10. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignmentcited by 41
  11. 2021When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfercited by 41
  12. 2022MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognitioncited by 38
  13. 2023OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretrainingcited by 37
  14. 2022Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Modelscited by 32
  15. 2023When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languagescited by 32
  16. 2021Multilingual Language Models Predict Human Reading Behaviorcited by 29
  17. 2022A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bankcited by 29
  18. 2022Pretraining with Artificial Language: Studying Transferable Knowledge in Language Modelscited by 28
  19. 2022Discovering Language-neutral Sub-networks in Multilingual Language Modelscited by 28
  20. 2022Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languagescited by 28
  21. 2023Zero-shot cross-lingual transfer language selection using linguistic similaritycited by 28
  22. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 27
  23. 2022Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferabilitycited by 26
  24. 2022Towards Best Practices for Training Multilingual Dense Retrieval Modelscited by 26
  25. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignmentcited by 26
  26. 2021An Isotropy Analysis in the Multilingual BERT Embedding Spacecited by 25
  27. 2022Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languagescited by 25
  28. 2022Insights into Pre-training via Simpler Synthetic Taskscited by 25
  29. 2021Predicting the Performance of Multilingual NLP Modelscited by 24
  30. 2021Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglotscited by 23
  31. 2021Multilingual Offensive Language Identification for Low-resource Languagescited by 22
  32. 2021On Cross-Lingual Retrieval with Multilingual Text Encoderscited by 22
  33. 2023Why Does Zero-Shot Cross-Lingual Generation Fail? An Explanation and a Solutioncited by 22
  34. 2021Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Trainingcited by 21
  35. 2021Specializing Multilingual Language Models: An Empirical Studycited by 21
  36. 2021Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapterscited by 21
  37. 2022LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translationcited by 21
  38. 2022Multi Task Learning For Zero Shot Performance Prediction of Multilingual Modelscited by 20

50 most cited of 182 All 182 (JSON)