mlsanityData to 7 Oct 2026

Cross-Linguistic Syntactic Difference in Multilingual BERT: How Good is It and How Does It Affect Transfer?

arXiv 2212.10879 ↗First posted 21 Dec 2022cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Multilingual BERT (mBERT) has demonstrated considerable cross-lingual syntactic ability, whereby it enables effective zero-shot cross-lingual transfer of syntactic knowledge. The transfer is more successful between some languages, but it is not well understood what leads to this variation and whether it fairly reflects difference between languages. In this work, we investigate the distributions of grammatical relations induced from mBERT in the context of 24 typologically different languages. We demonstrate that the distance between the distributions of different languages is highly consistent with the syntactic difference in terms of linguistic formalisms. Such difference learnt via self-supervision plays a crucial role in the zero-shot transfer performance and can be predicted by variation in morphosyntactic properties between languages. These results suggest that mBERT properly encodes languages in a way consistent with linguistic diversity and provide insights into the mechanism of cross-lingual transfer.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 34 of its 70 references that are in the map. First posted on arXiv 21 Dec 2022.

Builds on

34 of 70 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  4. 2019How multilingual is Multilingual BERT?cited by 642
  5. 2019Designing and Interpreting Probes with Control Taskscited by 608
  6. 2017Word Translation Without Parallel Datacited by 480
  7. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  8. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  9. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  10. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
  11. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  12. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
22 more
  1. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  2. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  3. 2013Exploiting Similarities among Languages for Machine Translationcited by 187
  4. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  5. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
  6. 2020Geometric Dataset Distances via Optimal Transportcited by 157
  7. 2019Choosing Transfer Languages for Cross-Lingual Learningcited by 143
  8. 2020Multilingual Alignment of Contextual Word Representationscited by 113
  9. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  10. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  11. 2020Syntactic Structure from Deep Learningcited by 99
  12. 2021First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERTcited by 64
  13. 2020Cross-Linguistic Syntactic Evaluation of Word Prediction Modelscited by 54
  14. 2021On the proper role of linguistically-oriented deep net analysis in linguistic theorizingcited by 43
  15. 2021A Closer Look at How Fine-tuning Changes BERTcited by 42
  16. 2021Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERTcited by 41
  17. 2020It's not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERTcited by 30
  18. 2016Many Languages, One Parsercited by 26
  19. 2020What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological propertiescited by 23
  20. 2014BilBOWA: Fast Bilingual Distributed Representations without Word Alignmentscited by 16
  21. 2021Analysing The Impact Of Linguistic Features On Cross-Lingual Transfercited by 16
  22. 2021Does Typological Blinding Impede Cross-Lingual Sharing?cited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 4; 2025: 1; 2026: 2.

10 papers.

  1. 2024Unveiling Linguistic Regions in Large Language Modelscited by 22
  2. 2024Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Modelscited by 11
  3. 2023CreoleVal: Multilingual Multitask Benchmarks for Creolescited by 9
  4. 2023Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalizationcited by 6
  5. 2024What is "Typological Diversity" in NLP?cited by 3
  6. 2024InstaTrans: An Instruction-Aware Translation Framework for Non-English Instruction Datasetscited by 2
  7. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  8. 2023Knowledge Trees: Gradient Boosting Decision Trees on Knowledge Neurons as Probing Classifiercited by 0
  9. 2025Multilingual BERT language model for medical tasks: Evaluation on domain-specific adaptation and cross-lingualitycited by 0
  10. 2026RunyaNER: Auxiliary Language Selection for Runyankore NERcited by 0