mlsanityData to 7 Oct 2026

Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure

arXiv 2203.08430 ↗First posted 16 Mar 2022cs.CLcited by 17

Not on the map. Why?

Abstract (arXiv)

Multilingual pre-trained language models, such as mBERT and XLM-R, have shown impressive cross-lingual ability. Surprisingly, both of them use multilingual masked language model (MLM) without any cross-lingual supervision or aligned data. Despite the encouraging results, we still lack a clear understanding of why cross-lingual ability could emerge from multilingual MLM. In our work, we argue that cross-language ability comes from the commonality between languages. Specifically, we study three language properties: constituent order, composition and word co-occurrence. First, we create an artificial language by modifying property in source language. Then we study the contribution of modified property through the change of cross-language transfer results on target language. We conduct experiments on six languages and two cross-lingual NLP tasks (textual entailment, sentence retrieval). Our main conclusion is that the contribution of constituent order and word co-occurrence is limited, while the composition is more crucial to the success of cross-linguistic transfer.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 26 of its 40 references that are in the map. First posted on arXiv 16 Mar 2022.

Builds on

26 of 40 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  6. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  7. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  8. 2019How multilingual is Multilingual BERT?cited by 642
  9. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  10. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  11. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  12. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
14 more
  1. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  2. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  3. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
  4. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  5. 2017A Survey Of Cross-lingual Word Embedding Modelscited by 142
  6. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  7. 2020On the Language Neutrality of Pre-trained Multilingual Representationscited by 82
  8. 2021First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERTcited by 64
  9. 2020ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corporacited by 63
  10. 2020On Learning Universal Representations Across Languagescited by 58
  11. 2020English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Toocited by 39
  12. 2019Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Informationcited by 37
  13. 2020VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generationcited by 34
  14. 2018Recursive Neural Network Based Preordering for English-to-Japanese Machine Translationcited by 3

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 8; 2024: 6; 2025: 2.

17 papers.

Sort:

  1. 2024How do Large Language Models Handle Multilingualism?cited by 148
  2. 2023OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretrainingcited by 37
  3. 2023When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languagescited by 32
  4. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 27
  5. 2024Multilingual Knowledge Editing with Language-Agnostic Factual Neuronscited by 18
  6. 2023Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Modelscited by 16
  7. 2025Language Models' Factuality Depends on the Language of Inquirycited by 14
  8. 2024Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Modelscited by 11
  9. 2022Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoderscited by 7
  10. 2023Machine-Created Universal Language for Cross-lingual Transfercited by 7
  11. 2023Self-Augmentation Improves Zero-Shot Cross-Lingual Transfercited by 3
  12. 2023Cross-Lingual Transfer of Cognitive Processing Complexitycited by 2
5 more
  1. 2025Merging Language and Domain Specific Models: The Impact on Technical Vocabulary Acquisitioncited by 2
  2. 2024Evaluating and explaining training strategies for zero-shot cross-lingual news sentiment analysiscited by 1
  3. 2023Measuring Cross-Lingual Transferability of Multilingual Transformers on Sentence Classificationcited by 0
  4. 2023LaDA: Latent Dialogue Action For Zero-shot Cross-lingual Neural Network Language Modelingcited by 0
  5. 2024Assessing the Role of Lexical Semantics in Cross-lingual Transfer through Controlled Manipulationscited by 0