# Language Embeddings Sometimes Contain Typological Generalizations

arXiv: https://arxiv.org/abs/2301.08115 · HTML: https://mlsanity.com/paper/2301.08115/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 94 references that are in the map.
- First posted on arXiv 19 Jan 2023.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [OpenNMT: Open-source Toolkit for Neural Machine Translation](https://mlsanity.com/paper/1709.03815/) (2017)
- [Systematic Inequalities in Language Technology Performance across the World's Languages](https://mlsanity.com/paper/2110.06733/) (2021)
- [Finding Universal Grammatical Relations in Multilingual BERT](https://mlsanity.com/paper/2005.04511/) (2020)
- [Offline bilingual word vectors, orthogonal transformations and the inverted softmax](https://mlsanity.com/paper/1702.03859/) (2017)
- [UDapter: Language Adaptation for Truly Universal Dependency Parsing](https://mlsanity.com/paper/2004.14327/) (2020)
- [Investigating Multilingual NMT Representations at Scale](https://mlsanity.com/paper/1909.02197/) (2019)
- [Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing](https://mlsanity.com/paper/1807.00914/) (2018)
- [Multilingual Neural Machine Translation with Language Clustering](https://mlsanity.com/paper/1908.09324/) (2019)
- [Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation](https://mlsanity.com/paper/2203.09435/) (2022)
- [Learning Language Representations for Typology Prediction](https://mlsanity.com/paper/1707.09569/) (2017)
- [Contextual Parameter Generation for Universal Neural Machine Translation](https://mlsanity.com/paper/1808.08493/) (2018)
- [Probing Multilingual BERT for Genetic and Typological Signals](https://mlsanity.com/paper/2011.02070/) (2020)
- [Many Languages, One Parser](https://mlsanity.com/paper/1602.01595/) (2016)
- [Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representations](https://mlsanity.com/paper/2004.14923/) (2020)
- [Continuous multilinguality with language vectors](https://mlsanity.com/paper/1612.07486/) (2016)
- [Past, Present, Future: A Computational Investigation of the Typology of Tense in 1000 Languages](https://mlsanity.com/paper/1704.08914/) (2017)
- [From Phonology to Syntax: Unsupervised Linguistic Typology at Different Levels with Language Embeddings](https://mlsanity.com/paper/1802.09375/) (2018)
- [Semantic Drift in Multilingual Representations](https://mlsanity.com/paper/1904.10820/) (2019)
- [SIGTYP 2020 Shared Task: Prediction of Typological Features](https://mlsanity.com/paper/2010.08246/) (2020)
- [Emerging Language Spaces Learned From Massively Multilingual Corpora](https://mlsanity.com/paper/1802.00273/) (2018)
- [Does Typological Blinding Impede Cross-Lingual Sharing?](https://mlsanity.com/paper/2101.11888/) (2021)
- [A Probabilistic Generative Model of Linguistic Typology](https://mlsanity.com/paper/1903.10950/) (2019)
- [Tracking Typological Traits of Uralic Languages in Distributed Language Representations](https://mlsanity.com/paper/1711.05468/) (2017)

## Built on by (most cited first, 9 of 9)

- [A Crosslingual Investigation of Conceptualization in 1335 Languages](https://mlsanity.com/paper/2305.08475/) (2023)
- [Crosslingual Transfer Learning for Low-Resource Languages Based on Multilingual Colexification Graphs](https://mlsanity.com/paper/2305.12818/) (2023)
- [Is Prompt-Based Finetuning Always Better than Vanilla Finetuning? Insights from Cross-Lingual Language Understanding](https://mlsanity.com/paper/2307.07880/) (2023)
- [Large Language Models are Easily Confused: A Quantitative Metric, Security Implications and Typological Analysis](https://mlsanity.com/paper/2410.13237/) (2024)
- [A study of conceptual language similarity: comparison and evaluation](https://mlsanity.com/paper/2305.13401/) (2023)
- [Colexifications for Bootstrapping Cross-lingual Datasets: The Case of Phonology, Concreteness, and Affectiveness](https://mlsanity.com/paper/2306.02646/) (2023)
- [The Medium Is Not the Message: Deconfounding Document Embeddings via Linear Concept Erasure](https://mlsanity.com/paper/2507.01234/) (2025)
- [Typological Feature Prediction with Large Language Models: An In-Context Learning Approach](https://mlsanity.com/paper/2609.03775/) (2026)
- [Self-Supervised Lexical Representation Learning for Fast, Large-Scale Phylogenetic Inference](https://mlsanity.com/paper/2609.05262/) (2026)

## Abstract (arXiv)

To what extent can neural network models learn generalizations about language structure, and how do we find out what they have learned? We explore these questions by training neural models for a range of natural language processing tasks on a massively multilingual dataset of Bible translations in 1295 languages. The learned language representations are then compared to existing typological databases as well as to a novel set of quantitative syntactic and morphological features obtained through annotation projection. We conclude that some generalizations are surprisingly close to traditional features from linguistic typology, but that most of our models, as well as those of previous work, do not appear to have made linguistically meaningful generalizations. Careful attention to details in the evaluation turns out to be essential to avoid false positives. Furthermore, to encourage continued work in this field, we release several resources covering most or all of the languages in our data: (i) multiple sets of language representations, (ii) multilingual word embeddings, (iii) projected and predicted syntactic and morphological features, (iv) software to provide linguistically sound evaluations of language representations.