mlsanityData to 14 Oct 2026

First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT

arXiv 2101.11109 ↗First posted 26 Jan 2021cs.CLcited by 65

Not on the map. Why?

Abstract (arXiv)

Multilingual pretrained language models have demonstrated remarkable zero-shot cross-lingual transfer capabilities. Such transfer emerges by fine-tuning on a task of interest in one language and evaluating on a distinct language, not seen during the fine-tuning. Despite promising results, we still lack a proper understanding of the source of this transfer. Using a novel layer ablation technique and analyses of the model's internal representations, we show that multilingual BERT, a popular multilingual language model, can be viewed as the stacking of two sub-networks: a multilingual encoder followed by a task-specific language-agnostic predictor. While the encoder is crucial for cross-lingual transfer and remains mostly unchanged during fine-tuning, the task predictor has little importance on the transfer and can be reinitialized during fine-tuning. We present extensive experiments with three distinct tasks, seventeen typologically diverse languages and multiple domains to support our hypothesis.

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). Builds on 22 of its 40 references that are in the map. First posted on arXiv 26 Jan 2021.

Builds on

22 of 40 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  4. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  6. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  7. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  8. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  9. 2019How multilingual is Multilingual BERT?cited by 645
  10. 2019Designing and Interpreting Probes with Control Taskscited by 640
  11. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  12. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
10 more
  1. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  2. 2013Exploiting Similarities among Languages for Machine Translationcited by 191
  3. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 181
  4. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  5. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 151
  6. 2017A Survey Of Cross-lingual Word Embedding Modelscited by 143
  7. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  8. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 110
  9. 2017Offline bilingual word vectors, orthogonal transformations and the inverted softmaxcited by 96
  10. 2020It's not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERTcited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 13; 2023: 15; 2024: 14; 2025: 10; 2026: 8.

65 papers.

Sort:

By area:

  1. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  2. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  3. 2024Probing the Emergence of Cross-lingual Alignment during LLM Trainingcited by 39
  4. 2022Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Modelscited by 32
  5. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 29
  6. 2022Discovering Language-neutral Sub-networks in Multilingual Language Modelscited by 28
  7. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 28
  8. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  9. 2024Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelscited by 21
  10. 2025Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipelinecited by 21
  11. 2024Discovering Low-rank Subspaces for Language-agnostic Multilingual Representationscited by 19
  12. 2022Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structurecited by 17
38 more
  1. 2024The Hidden Space of Transformer Language Adapterscited by 17
  2. 2024Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Modelscited by 15
  3. 2022Does Transliteration Help Multilingual Language Modeling?cited by 14
  4. 2023How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuningcited by 14
  5. 2023MELA: Multilingual Evaluation of Linguistic Acceptabilitycited by 14
  6. 2024Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledgecited by 14
  7. 2025Language Models' Factuality Depends on the Language of Inquirycited by 14
  8. 2023Identifying the Correlation Between Language Distance and Cross-Lingual Transfer in a Multilingual Representation Spacecited by 12
  9. 2021Can Character-based Language Models Improve Downstream Task Performance in Low-Resource and Noisy Language Scenarios?cited by 11
  10. 2022Languages You Know Influence Those You Learn: Impact of Language Characteristics on Multi-Lingual Text-to-Text Transfercited by 11
  11. 2024Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Modelscited by 11
  12. 2021Multilingual Counter Narrative Type Classificationcited by 10
  13. 2022Cross-Linguistic Syntactic Difference in Multilingual BERT: How Good is It and How Does It Affect Transfer?cited by 10
  14. 2025The Translation Barrier Hypothesis: Multilingual Generation with Large Language Models Suffers from Implicit Translation Failurecited by 10
  15. 2025Beyond Literal Token Overlap: Token Alignability for Multilingualitycited by 9
  16. 2022Combining Static and Contextualised Multilingual Embeddingscited by 8
  17. 2022Cross-lingual Similarity of Multilingual Representations Revisitedcited by 8
  18. 2022WIDER & CLOSER: Mixture of Short-channel Distillers for Zero-shot Cross-lingual Named Entity Recognitioncited by 7
  19. 2024Extracting Sentence Embeddings from Pretrained Transformer Modelscited by 7
  20. 2021Do Multilingual Neural Machine Translation Models Contain Language Pair Specific Attention Heads?cited by 6
  21. 2021Similarity of Sentence Representations in Multilingual LMs: Resolving Conflicting Literature and Case Study of Baltic Languagescited by 6
  22. 2021Locating Language-Specific Information in Contextualized Embeddingscited by 6
  23. 2023Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalizationcited by 6
  24. 2024Mitigating the Linguistic Gap with Phonemic Representations for Robust Cross-lingual Transfercited by 6
  25. 2025Large Language Models as Psychological Simulators: A Methodological Guidecited by 6
  26. 2023A Joint Matrix Factorization Analysis of Multilingual Representationscited by 5
  27. 2024Analysis of Multi-Source Language Training in Cross-Lingual Transfercited by 5
  28. 2022Multi-Level Contrastive Learning for Cross-Lingual Alignmentcited by 4
  29. 2023ContraSim -- Analyzing Neural Representations Based on Contrastive Learningcited by 4
  30. 2022OneAligner: Zero-shot Cross-lingual Transfer with One Rich-Resource Language Pair for Low-Resource Sentence Retrievalcited by 3
  31. 2024Understanding the role of FFNs in driving multilingual behaviour in LLMscited by 3
  32. 2025Beyond the Rosetta Stone: Unification Forces in Generalization Dynamicscited by 3
  33. 2025Controlling the Risk of Corrupted Contexts for Language Models via Early-Exitingcited by 3
  34. 2025Model-Based Ranking of Source Languages for Zero-Shot Cross-Lingual Transfercited by 3
  35. 2026Can Embedding Similarity Predict Cross-Lingual Transfer? A Systematic Study on African Languagescited by 3
  36. 2023In What Languages are Generative Language Models the Most Formal? Analyzing Formality Distribution across Languagescited by 2
  37. 2023Comparing Styles across Languages: A Cross-Cultural Exploration of Politenesscited by 2
  38. 2025High-Dimensional Interlingual Representations of Large Language Modelscited by 2

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Exploring the Relationship between Alignment and Cross-lingual Transfer in Multilingual Transformers
  2. 2023Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review
  3. 2025Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders
  4. 2025Cross-Lingual Generalization and Compression: From Language-Specific to Shared Neurons
  5. 2025A Post-trainer's Guide to Multilingual Training Data: Uncovering Cross-lingual Transfer Dynamics
  6. 2021On the Language-specificity of Multilingual BERT and the Impact of Fine-tuning
  7. 2024Understanding Cross-Lingual Alignment -- A Survey
  8. 2020Multilingual BERT Post-Pretraining Alignment