# What do Language Representations Really Represent?

arXiv: https://arxiv.org/abs/1901.02646 · HTML: https://mlsanity.com/paper/1901.02646/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jan 2019.

## Built on by (most cited first, 17 of 17)

- [Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge Distillation](https://mlsanity.com/paper/2112.03473/) (2021)
- [Comparing Feature-Engineering and Feature-Learning Approaches for Multilingual Translationese Classification](https://mlsanity.com/paper/2109.07604/) (2021)
- [Does Typological Blinding Impede Cross-Lingual Sharing?](https://mlsanity.com/paper/2101.11888/) (2021)
- [Language Embeddings for Typology and Cross-lingual Transfer Learning](https://mlsanity.com/paper/2106.02082/) (2021)
- [How Familiar Does That Sound? Cross-Lingual Representational Similarity Analysis of Acoustic Word Embeddings](https://mlsanity.com/paper/2109.10179/) (2021)
- [Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders](https://mlsanity.com/paper/2205.00267/) (2022)
- [The Past, Present, and Future of Typological Databases in NLP](https://mlsanity.com/paper/2310.13440/) (2023)
- [Large Language Models are Easily Confused: A Quantitative Metric, Security Implications and Typological Analysis](https://mlsanity.com/paper/2410.13237/) (2024)
- [The Shape of Word Embeddings: Quantifying Non-Isometry With Topological Data Analysis](https://mlsanity.com/paper/2404.00500/) (2024)
- [Neighbors and relatives: How do speech embeddings reflect linguistic connections across the world?](https://mlsanity.com/paper/2506.08564/) (2025)
- [Multilingual Neural Machine Translation:Can Linguistic Hierarchies Help?](https://mlsanity.com/paper/2110.07816/) (2021)
- [Cross-Lingual Transfer of Cognitive Processing Complexity](https://mlsanity.com/paper/2302.12695/) (2023)
- [Colexifications for Bootstrapping Cross-lingual Datasets: The Case of Phonology, Concreteness, and Affectiveness](https://mlsanity.com/paper/2306.02646/) (2023)
- [Informative Language Representation Learning for Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/2209.01530/) (2022)
- [LCS: A Language Converter Strategy for Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/2406.02876/) (2024)
- [Parameter-efficient Adaptation of Multilingual Multimodal Models for Low-resource ASR](https://mlsanity.com/paper/2410.13445/) (2024)
- [Self-Supervised Lexical Representation Learning for Fast, Large-Scale Phylogenetic Inference](https://mlsanity.com/paper/2609.05262/) (2026)

## Abstract (arXiv)

A neural language model trained on a text corpus can be used to induce distributed representations of words, such that similar words end up with similar representations. If the corpus is multilingual, the same model can be used to learn distributed representations of languages, such that similar languages end up with similar representations. We show that this holds even when the multilingual corpus has been translated into English, by picking up the faint signal left by the source languages. However, just like it is a thorny problem to separate semantic from syntactic similarity in word representations, it is not obvious what type of similarity is captured by language representations. We investigate correlations and causal relationships between language representations learned from translations on one hand, and genetic, geographical, and several levels of structural similarity between languages on the other. Of these, structural similarity is found to correlate most strongly with language representation similarity, while genetic relationships---a convenient benchmark used for evaluation in previous work---appears to be a confounding factor. Apart from implications about translation effects, we see this more generally as a case where NLP and linguistic typology can interact and benefit one another.