mlsanityData to 14 Oct 2026

Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT

arXiv 1904.09077 ↗First posted 19 Apr 2019cs.CLcited by 301

Not on the map. Why?

Abstract (arXiv)

Pretrained contextual representation models (Peters et al., 2018; Devlin et al., 2018) have pushed forward the state-of-the-art on many NLP tasks. A new release of BERT (Devlin, 2018) includes a model simultaneously pretrained on 104 languages with impressive performance for zero-shot cross-lingual transfer on a natural language inference task. This paper explores the broader cross-lingual potential of mBERT (multilingual) as a zero shot language transfer model on 5 NLP tasks covering a total of 39 languages from various language families: NLI, document classification, NER, POS tagging, and dependency parsing. We compare mBERT with the best-published methods for zero-shot cross-lingual transfer and find mBERT competitive on each task. Additionally, we investigate the most effective strategy for utilizing mBERT in this manner, determine to what extent mBERT generalizes away from language specific features, and measure factors that influence cross-lingual transfer.

Key facts. Cited by 301 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 93; 2022: 64; 2023: 55; 2024: 40; 2025: 29; 2026: 20.

301 papers.

Sort:

By area:

  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  2. 2023BloombergGPT: A Large Language Model for Financecited by 939
  3. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  4. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  5. 2022Challenges and Strategies in Cross-Cultural NLPcited by 218
  6. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  7. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  8. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  9. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 132
  10. 2021The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Modelscited by 119
  11. 2021mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Datasetcited by 116
  12. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
38 more
  1. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  2. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  3. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 95
  4. 2021X-FACT: A New Benchmark Dataset for Multilingual Fact Checkingcited by 91
  5. 2023Spanish Pre-trained BERT Model and Evaluation Datacited by 81
  6. 2024Multilingual Instruction Tuning With Just a Pinch of Multilingualitycited by 73
  7. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  8. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  9. 2021First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERTcited by 65
  10. 2021Are Multilingual Models Effective in Code-Switching?cited by 56
  11. 2024ALLaM: Large Language Models for Arabic and Englishcited by 55
  12. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  13. 2021AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languagescited by 50
  14. 2023Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Reviewcited by 50
  15. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  16. 2021Revisiting the Primacy of English in Zero-shot Cross-lingual Transfercited by 47
  17. 2022Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptationcited by 44
  18. 2021When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfercited by 42
  19. 2021Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERTcited by 41
  20. 2021Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Modelscited by 41
  21. 2021MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairscited by 41
  22. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignmentcited by 41
  23. 2021Multi-view Subword Regularizationcited by 40
  24. 2023Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?cited by 39
  25. 2024Probing the Emergence of Cross-lingual Alignment during LLM Trainingcited by 39
  26. 2022MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognitioncited by 38
  27. 2023Lost in Translation: Large Language Models in Non-English Content Analysiscited by 35
  28. 2022Pretraining with Artificial Language: Studying Transferable Knowledge in Language Modelscited by 28
  29. 2022Discovering Language-neutral Sub-networks in Multilingual Language Modelscited by 28
  30. 2022Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languagescited by 28
  31. 2023Zero-shot cross-lingual transfer language selection using linguistic similaritycited by 28
  32. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 28
  33. 2023Language Versatilists vs. Specialists: An Empirical Revisiting on Multilingual Transfer Abilitycited by 27
  34. 2022Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferabilitycited by 26
  35. 2022Towards Best Practices for Training Multilingual Dense Retrieval Modelscited by 26
  36. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignmentcited by 26
  37. 2024Data Contamination Can Cross Language Barrierscited by 26
  38. 2021An Isotropy Analysis in the Multilingual BERT Embedding Spacecited by 25

50 most cited of 301 All 301 (JSON)