# What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological properties

arXiv: https://arxiv.org/abs/2009.12862 · HTML: https://mlsanity.com/paper/2009.12862/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Sep 2020.

## Built on by (most cited first, 23 of 23)

- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [The Geometry of Multilingual Language Model Representations](https://mlsanity.com/paper/2205.10964/) (2022)
- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers](https://mlsanity.com/paper/2411.08745/) (2024)
- [Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages](https://mlsanity.com/paper/2203.14139/) (2022)
- [Discovering Language-neutral Sub-networks in Multilingual Language Models](https://mlsanity.com/paper/2205.12672/) (2022)
- [Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids' Representations](https://mlsanity.com/paper/2109.05958/) (2021)
- [Cross-Linguistic Syntactic Difference in Multilingual BERT: How Good is It and How Does It Affect Transfer?](https://mlsanity.com/paper/2212.10879/) (2022)
- [Does Typological Blinding Impede Cross-Lingual Sharing?](https://mlsanity.com/paper/2101.11888/) (2021)
- [On the Language-specificity of Multilingual BERT and the Impact of Fine-tuning](https://mlsanity.com/paper/2109.06935/) (2021)
- [Are Pretrained Multilingual Models Equally Fair Across Languages?](https://mlsanity.com/paper/2210.05457/) (2022)
- [Locating Language-Specific Information in Contextualized Embeddings](https://mlsanity.com/paper/2109.08040/) (2021)
- [A Joint Matrix Factorization Analysis of Multilingual Representations](https://mlsanity.com/paper/2310.15513/) (2023)
- [Analysis of Multi-Source Language Training in Cross-Lingual Transfer](https://mlsanity.com/paper/2402.13562/) (2024)
- [RuSentEval: Linguistic Source, Encoder Force!](https://mlsanity.com/paper/2103.00573/) (2021)
- [Discovering Representation Sprachbund For Multilingual Pre-Training](https://mlsanity.com/paper/2109.00271/) (2021)
- [Multilingual Transformer Encoders: a Word-Level Task-Agnostic Evaluation](https://mlsanity.com/paper/2207.09076/) (2022)
- [The Less the Merrier? Investigating Language Representation in Multilingual Models](https://mlsanity.com/paper/2310.13228/) (2023)
- [The Grammar and Syntax Based Corpus Analysis Tool For The Ukrainian Language](https://mlsanity.com/paper/2305.13530/) (2023)
- [Differential Privacy, Linguistic Fairness, and Training Data Influence: Impossibility and Possibility Theorems for Multilingual Language Models](https://mlsanity.com/paper/2308.08774/) (2023)
- [Do Multi-Lingual Pre-trained Language Models Reveal Consistent Token Attributions in Different Languages?](https://mlsanity.com/paper/2112.12356/) (2021)
- [The correlation between nativelike selection and prototypicality: a multilingual onomasiological case study using semantic embedding](https://mlsanity.com/paper/2405.13529/) (2024)
- [When Language Representations Interact: Separability and Cross-Lingual Effects in LLMs](https://mlsanity.com/paper/2606.14347/) (2026)

## Abstract (arXiv)

Multilingual sentence encoders have seen much success in cross-lingual model transfer for downstream NLP tasks. Yet, we know relatively little about the properties of individual languages or the general patterns of linguistic variation that they encode. We propose methods for probing sentence representations from state-of-the-art multilingual encoders (LASER, M-BERT, XLM and XLM-R) with respect to a range of typological properties pertaining to lexical, morphological and syntactic structure. In addition, we investigate how this information is distributed across all layers of the models. Our results show interesting differences in encoding linguistic variation associated with different pretraining strategies.