# Probing Multilingual BERT for Genetic and Typological Signals

arXiv: https://arxiv.org/abs/2011.02070 · HTML: https://mlsanity.com/paper/2011.02070/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Nov 2020.

## Built on by (most cited first, 27 of 27)

- [The Geometry of Multilingual Language Model Representations](https://mlsanity.com/paper/2205.10964/) (2022)
- [Phylogeny-Inspired Adaptation of Multilingual Models to New Languages](https://mlsanity.com/paper/2205.09634/) (2022)
- [Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languages](https://mlsanity.com/paper/2212.09651/) (2022)
- [Multilingual Byte2Speech Models for Scalable Low-resource Speech Synthesis](https://mlsanity.com/paper/2103.03541/) (2021)
- [Languages You Know Influence Those You Learn: Impact of Language Characteristics on Multi-Lingual Text-to-Text Transfer](https://mlsanity.com/paper/2212.01757/) (2022)
- [The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?](https://mlsanity.com/paper/2601.07220/) (2026)
- [Language Embeddings Sometimes Contain Typological Generalizations](https://mlsanity.com/paper/2301.08115/) (2023)
- [Natural Language Processing RELIES on Linguistics](https://mlsanity.com/paper/2405.05966/) (2024)
- [On the Language-specificity of Multilingual BERT and the Impact of Fine-tuning](https://mlsanity.com/paper/2109.06935/) (2021)
- [Are Pretrained Multilingual Models Equally Fair Across Languages?](https://mlsanity.com/paper/2210.05457/) (2022)
- [Linguistic Interpretability of Transformer-based Language Models: a systematic review](https://mlsanity.com/paper/2504.08001/) (2025)
- [Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages](https://mlsanity.com/paper/2305.02215/) (2023)
- [Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Models](https://mlsanity.com/paper/2403.12809/) (2024)
- [A study of conceptual language similarity: comparison and evaluation](https://mlsanity.com/paper/2305.13401/) (2023)
- [Towards a Deep Understanding of Multilingual End-to-End Speech Translation](https://mlsanity.com/paper/2310.20456/) (2023)
- [Multilingual Language Models Encode Script Over Linguistic Structure](https://mlsanity.com/paper/2604.05090/) (2026)
- [Cross-Lingual Transfer of Cognitive Processing Complexity](https://mlsanity.com/paper/2302.12695/) (2023)
- [Do Multi-Lingual Pre-trained Language Models Reveal Consistent Token Attributions in Different Languages?](https://mlsanity.com/paper/2112.12356/) (2021)
- [Can the Language of the Collation be Translated into the Language of the Stemma? Using Machine Translation for Witness Localization](https://mlsanity.com/paper/2206.05603/) (2022)
- [Relationship of the language distance to English ability of a country](https://mlsanity.com/paper/2211.07855/) (2022)
- [mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2305.13684/) (2023)
- [Deep Language Geometry: Constructing a Metric Space from LLM Weights](https://mlsanity.com/paper/2508.11676/) (2025)
- [Harnessing Linguistic Dissimilarity for Language Generalization on Unseen Low-Resource Varieties](https://mlsanity.com/paper/2605.04500/) (2026)
- [Similarity All The Way Up: Multilingual Generalization in LLMs Relies on Language-Level Similarity Structures](https://mlsanity.com/paper/2607.22699/) (2026)
- [Self-Supervised Lexical Representation Learning for Fast, Large-Scale Phylogenetic Inference](https://mlsanity.com/paper/2609.05262/) (2026)
- [Quantitative Measurement of Language Distance among Closely Related Indo-European Languages Using Pretrained Language Models: A Case Study on the North Germanic Branch](https://mlsanity.com/paper/2609.34152/) (2026)
- [QuanLing: Cross-Branch Validation of Language Distance Quantification on Western Romance](https://mlsanity.com/paper/2610.08851/) (2026)

## Abstract (arXiv)

We probe the layers in multilingual BERT (mBERT) for phylogenetic and geographic language signals across 100 languages and compute language distances based on the mBERT representations. We 1) employ the language distances to infer and evaluate language trees, finding that they are close to the reference family tree in terms of quartet tree distance, 2) perform distance matrix regression analysis, finding that the language distances can be best explained by phylogenetic and worst by structural factors and 3) present a novel measure for measuring diachronic meaning stability (based on cross-lingual representation variability) which correlates significantly with published ranked lists based on linguistic approaches. Our results contribute to the nascent field of typological interpretability of cross-lingual text representations.