# High-Dimensional Interlingual Representations of Large Language Models

arXiv: https://arxiv.org/abs/2503.11280 · HTML: https://mlsanity.com/paper/2503.11280/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 70 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Builds on (most cited first, 30 of 33)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [Exploiting Similarities among Languages for Machine Translation](https://mlsanity.com/paper/1309.4168/) (2013)
- [Relative representations enable zero-shot latent space communication](https://mlsanity.com/paper/2209.15430/) (2022)
- [Emerging Cross-lingual Structure in Pretrained Language Models](https://mlsanity.com/paper/1911.01464/) (2019)
- [Continual Lifelong Learning in Natural Language Processing: A Survey](https://mlsanity.com/paper/2012.09823/) (2020)
- [How do Large Language Models Handle Multilingualism?](https://mlsanity.com/paper/2402.18815/) (2024)
- [Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier](https://mlsanity.com/paper/2412.04261/) (2024)
- [The Geometry of Multilingual Language Model Representations](https://mlsanity.com/paper/2205.10964/) (2022)
- [On the Language Neutrality of Pre-trained Multilingual Representations](https://mlsanity.com/paper/2004.05160/) (2020)
- [LLM Internal States Reveal Hallucination Risk Faced With a Query](https://mlsanity.com/paper/2407.03282/) (2024)
- [LLMs Are Few-Shot In-Context Low-Resource Language Learners](https://mlsanity.com/paper/2403.16512/) (2024)
- [First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT](https://mlsanity.com/paper/2101.11109/) (2021)
- [On Learning Universal Representations Across Languages](https://mlsanity.com/paper/2007.15960/) (2020)
- [Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review](https://mlsanity.com/paper/2305.16768/) (2023)
- [Question Translation Training for Better Multilingual Reasoning](https://mlsanity.com/paper/2401.07817/) (2024)
- [A neural interlingua for multilingual machine translation](https://mlsanity.com/paper/1804.08198/) (2018)
- [Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning](https://mlsanity.com/paper/2305.16252/) (2023)
- [On the Universal Truthfulness Hyperplane Inside LLMs](https://mlsanity.com/paper/2407.08582/) (2024)
- [Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Models](https://mlsanity.com/paper/2410.11718/) (2024)
- [Consistency by Agreement in Zero-shot Neural Machine Translation](https://mlsanity.com/paper/1904.02338/) (2019)

## Built on by (most cited first, 2 of 2)

- [Can you map it to English? The Role of Cross-Lingual Alignment in Multilingual Performance of LLMs](https://mlsanity.com/paper/2504.09378/) (2025)
- [A Systematic Comparison of Multilingual Interpretability Methods Reveals Anisotropy-Driven Failures](https://mlsanity.com/paper/2609.04819/) (2026)

## Abstract (arXiv)

Large language models (LLMs) trained on massive multilingual datasets hint at the formation of interlingual constructs--a shared subspace in the representation space. However, evidence regarding this phenomenon is mixed, leaving it unclear whether these models truly develop unified interlingual representations, or present a partially aligned constructs. We explore 31 diverse languages varying on their resource-levels, typologies, and geographical regions; and find that multilingual LLMs exhibit inconsistent cross-lingual alignments. To address this, we propose an interlingual representation framework identifying both the shared interlingual semantic subspace and fragmented components, existed due to representational limitations. We introduce Interlingual Local Overlap (ILO) score to quantify interlingual alignment by comparing the local neighborhood structures of high-dimensional representations. We utilize ILO to investigate the impact of single-language fine-tuning on the interlingual representations in multilingual LLMs. Our results indicate that training exclusively on a single language disrupts the alignment in early layers, while freezing these layers preserves the alignment of interlingual representations, leading to improved cross-lingual generalization. These results validate our framework and metric for evaluating interlingual representation, and further underscore that interlingual alignment is crucial for scalable multilingual learning.