mlsanityData to 14 Oct 2026

Finding Universal Grammatical Relations in Multilingual BERT

arXiv 2005.04511 ↗First posted 9 May 2020cs.CL cs.LGcited by 110

Not on the map. Why?

Abstract (arXiv)

Recent work has found evidence that Multilingual BERT (mBERT), a transformer-based multilingual masked language model, is capable of zero-shot cross-lingual transfer, suggesting that some aspects of its representations are shared cross-lingually. To better understand this overlap, we extend recent work on finding syntactic trees in neural networks' internal representations to the multilingual setting. We show that subspaces of mBERT representations recover syntactic tree distances in languages other than English, and that these subspaces are approximately shared across languages. Motivated by these results, we present an unsupervised analysis method that provides evidence mBERT learns representations of syntactic dependency labels, in the form of clusters which largely agree with the Universal Dependencies taxonomy. This evidence suggests that even without explicit supervision, multilingual masked language models learn certain linguistic universals.

Key facts. Cited by 110 papers in the map (data through 14 Oct 2026). First posted on arXiv 9 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 24; 2022: 26; 2023: 20; 2024: 17; 2025: 11; 2026: 12.

110 papers.

Sort:

By area:

  1. 2023Large Language Modelscited by 780
  2. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  3. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 131
  4. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 130
  5. 2021The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Modelscited by 119
  6. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 103
  7. 2022The Geometry of Multilingual Language Model Representationscited by 95
  8. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 95
  9. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 71
  10. 2021It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoningcited by 69
  11. 2021First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERTcited by 65
  12. 2021Codified audio language modeling learns useful representations for music information retrievalcited by 64
38 more
  1. 2025Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languagescited by 59
  2. 2023AttentionViz: A Global View of Transformer Attentioncited by 50
  3. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  4. 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformerscited by 44
  5. 2021Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERTcited by 41
  6. 2022Models and Datasets for Cross-Lingual Summarisationcited by 36
  7. 2023Lost in Translation: Large Language Models in Non-English Content Analysiscited by 35
  8. 2022Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Modelscited by 32
  9. 2021Exploring the Role of BERT Token Representations to Explain Sentence Probing Resultscited by 29
  10. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 29
  11. 2022Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Modelscited by 28
  12. 2022Pretraining with Artificial Language: Studying Transferable Knowledge in Language Modelscited by 28
  13. 2022Discovering Language-neutral Sub-networks in Multilingual Language Modelscited by 28
  14. 2022Insights into Pre-training via Simpler Synthetic Taskscited by 26
  15. 2021An Isotropy Analysis in the Multilingual BERT Embedding Spacecited by 25
  16. 2024A polar coordinate system represents syntax in large language modelscited by 25
  17. 2024On the Similarity of Circuits across Languages: a Case Study on the Subject-verb Agreement Taskcited by 24
  18. 2024Relational Composition in Neural Networks: A Survey and Call to Actioncited by 23
  19. 2021Attention Can Reflect Syntactic Structure (If You Let It)cited by 22
  20. 2024Unveiling Linguistic Regions in Large Language Modelscited by 22
  21. 2021How much pretraining data do language models need to learn syntax?cited by 21
  22. 2023Discovering Universal Geometry in Embeddings with ICAcited by 19
  23. 2024Discovering Low-rank Subspaces for Language-agnostic Multilingual Representationscited by 19
  24. 2023Uncovering hidden geometry in Transformers via disentangling position and contextcited by 18
  25. 2022AST-Probe: Recovering abstract syntax trees from hidden representations of pre-trained language modelscited by 16
  26. 2023Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Modelscited by 16
  27. 2022Intriguing Properties of Compression on Multilingual Modelscited by 15
  28. 2023The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metricscited by 15
  29. 2021Cross-lingual Transfer of Monolingual Modelscited by 14
  30. 2022Causal Analysis of Syntactic Agreement Neurons in Multilingual Language Modelscited by 14
  31. 2023How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuningcited by 14
  32. 2024Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledgecited by 14
  33. 2021Micromodels for Efficient, Explainable, and Reusable Systems: A Case Study on Mental Healthcited by 13
  34. 2021gaBERT -- an Irish Language Modelcited by 12
  35. 2023The System Model and the User Model: Exploring AI Dashboard Designcited by 12
  36. 2023Investigating semantic subspaces of Transformer sentence embeddings through linear structural probingcited by 12
  37. 2024Probing Multimodal Large Language Models for Global and Local Semantic Representationscited by 12
  38. 2021Syntax-augmented Multilingual BERT for Cross-lingual Transfercited by 11

50 most cited of 110 All 110 (JSON)