mlsanityData to 14 Oct 2026

When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer

arXiv 2110.14782 ↗First posted 27 Oct 2021cs.CL cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

While recent work on multilingual language models has demonstrated their capacity for cross-lingual zero-shot transfer on downstream tasks, there is a lack of consensus in the community as to what shared properties between languages enable such transfer. Analyses involving pairs of natural languages are often inconclusive and contradictory since languages simultaneously differ in many linguistic aspects. In this paper, we perform a large-scale empirical study to isolate the effects of various linguistic properties by measuring zero-shot transfer between four diverse natural languages and their counterparts constructed by modifying aspects such as the script, word order, and syntax. Among other things, our experiments show that the absence of sub-word overlap significantly affects zero-shot transfer when languages differ in their word order, and there is a strong correlation between transfer performance and word embedding alignment between languages (e.g., R=0.94 on the task of NLI). Our results call for focus in multilingual models on explicitly improving word embedding alignment between languages rather than relying on its implicit emergence.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 27 Oct 2021.

Builds on

22 of 30 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  7. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  8. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  9. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 723
  10. 2019How multilingual is Multilingual BERT?cited by 645
  11. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  12. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
10 more
  1. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  2. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 181
  3. 2019Choosing Transfer Languages for Cross-Lingual Learningcited by 144
  4. 2020Multilingual Alignment of Contextual Word Representationscited by 114
  5. 2020Explicit Alignment Objectives for Multilingual Bidirectional Encoderscited by 50
  6. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  7. 2020DICT-MLM: Improved Multilingual Pre-Training using Bilingual Dictionariescited by 20
  8. 2017The Galactic Dependencies Treebanks: Getting More Data by Synthesizing New Languagescited by 19
  9. 2021Exploiting Language Relatedness for Low Web-Resource Language Model Adaptation: An Indic Languages Studycited by 19
  10. 2019Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Modelcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 8; 2023: 6; 2024: 9; 2025: 11; 2026: 8.

42 papers.

Sort:

By area:

  1. 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 126
  2. 2023Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Reviewcited by 50
  3. 2025Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biasescited by 37
  4. 2022Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Modelscited by 32
  5. 2024mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models?cited by 21
  6. 2022Intriguing Properties of Compression on Multilingual Modelscited by 15
  7. 2024Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignmentcited by 14
  8. 2023Unveiling Multilinguality in Transformer Models: Exploring Language Specificity in Feed-Forward Networkscited by 13
  9. 2024Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Modelscited by 11
  10. 2025False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Modelscited by 11
  11. 2022Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studiescited by 9
  12. 2025Beyond Literal Token Overlap: Token Alignability for Multilingualitycited by 9
30 more
  1. 2023Second Language Acquisition of Neural Language Modelscited by 8
  2. 2025Lugha-Llama: Adapting Large Language Models for African Languagescited by 8
  3. 2025Do You Know About My Nation? Investigating Multilingual Language Models' Cultural Literacy Through Factual Knowledgecited by 8
  4. 2022Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoderscited by 7
  5. 2022Ancestor-to-Creole Transfer is Not a Walk in the Parkcited by 7
  6. 2024An Efficient Approach for Studying Cross-Lingual Transfer in Multilingual Language Modelscited by 7
  7. 2022ALIGN-MLM: Word Embedding Alignment is Crucial for Multilingual Pre-trainingcited by 6
  8. 2024Pre-Trained Language Models Represent Some Geographic Populations Better Than Otherscited by 6
  9. 2024We're Calling an Intervention: Exploring Fundamental Hurdles in Adapting Language Models to Nonstandard Textcited by 6
  10. 2025Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluatorscited by 6
  11. 2022A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learningcited by 4
  12. 2024Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guidecited by 4
  13. 2023Does mBERT understand Romansh? Evaluating word embeddings using word alignmentcited by 3
  14. 2025Multilingual LLMs Struggle to Link Orthography and Semantics in Bilingual Word Processingcited by 3
  15. 2025Model-Based Ranking of Source Languages for Zero-Shot Cross-Lingual Transfercited by 3
  16. 2023Cross-Lingual Transfer of Cognitive Processing Complexitycited by 2
  17. 2025Merging Language and Domain Specific Models: The Impact on Technical Vocabulary Acquisitioncited by 2
  18. 2022MonoByte: A Pool of Monolingual Byte-level Language Modelscited by 1
  19. 2023DiTTO: A Feature Representation Imitation Approach for Improving Cross-Lingual Transfercited by 1
  20. 2025Enhancing Cross-Lingual Transfer through Reversible Transliteration: A Huffman-Based Approach for Low-Resource Languagescited by 1
  21. 2026Quantifying Cross-Lingual Transfer in Paralinguistic Speech Taskscited by 1
  22. 2024Assessing the Role of Lexical Semantics in Cross-lingual Transfer through Controlled Manipulationscited by 0
  23. 2025Multilingual BERT language model for medical tasks: Evaluation on domain-specific adaptation and cross-lingualitycited by 0
  24. 2026A Study of Crosslinguistic Influence in Language Modelscited by 0
  25. 2026Enhancing Multilingual Embeddings via Multi-Way Parallel Text Alignmentcited by 0
  26. 2026The Impact of Vocabulary Overlaps on Knowledge Transfer in Multilingual Machine Translationcited by 0
  27. 2026Improving Cross-Lingual Transfer for Sequential Sentence Classification in Research Papers via Structural Similaritycited by 0
  28. 2026Multilinguality in Hybrid Attention LLMscited by 0
  29. 2026RunyaNER: Auxiliary Language Selection for Runyankore NERcited by 0
  30. 2026Dataset-Specific Encoding of Image Content Underlies Cross-Dataset Failure in Brain MRI Foundation Modelscited by 0