# Concept Space Alignment in Multilingual LLMs

arXiv: https://arxiv.org/abs/2410.01079 · HTML: https://mlsanity.com/paper/2410.01079/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 40 references that are in the map.
- First posted on arXiv 1 Oct 2024.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models](https://mlsanity.com/paper/2310.10378/) (2023)
- [HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization](https://mlsanity.com/paper/2402.16694/) (2024)
- [On the Limitations of Unsupervised Bilingual Dictionary Induction](https://mlsanity.com/paper/1805.03620/) (2018)
- [Explicit Alignment Objectives for Multilingual Bidirectional Encoders](https://mlsanity.com/paper/2010.07972/) (2020)
- [xCoT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoning](https://mlsanity.com/paper/2401.07037/) (2024)
- [Multilingual BERT Post-Pretraining Alignment](https://mlsanity.com/paper/2010.12547/) (2020)
- [PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignment](https://mlsanity.com/paper/2407.16222/) (2024)
- [Language Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models?](https://mlsanity.com/paper/2311.03788/) (2023)
- [Do language models have coherent mental models of everyday things?](https://mlsanity.com/paper/2212.10029/) (2022)
- [Lost in Evaluation: Misleading Benchmarks for Bilingual Dictionary Induction](https://mlsanity.com/paper/1909.05708/) (2019)
- [A Discriminative Latent-Variable Model for Bilingual Lexicon Induction](https://mlsanity.com/paper/1808.09334/) (2018)
- [On the Tip of the Tongue: Analyzing Conceptual Representation in Large Language Models with Reverse-Dictionary Probe](https://mlsanity.com/paper/2402.14404/) (2024)
- [Limitations of Cross-Lingual Learning from Image Search](https://mlsanity.com/paper/1709.05914/) (2017)

## Built on by (most cited first, 13 of 13)

- [Multilingual Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2411.11072/) (2024)
- [Hypothesis-Driven Feature Manifold Analysis in LLMs via Supervised Multi-Dimensional Scaling](https://mlsanity.com/paper/2510.01025/) (2025)
- [Differential syntactic and semantic encoding in LLMs](https://mlsanity.com/paper/2601.04765/) (2026)
- [When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Training](https://mlsanity.com/paper/2601.22851/) (2026)
- [Tomato, Tomahto, Tomate: Do Multilingual Language Models Understand Based on Subword-Level Semantic Concepts?](https://mlsanity.com/paper/2411.04530/) (2024)
- [Probing LLMs for Multilingual Discourse Generalization Through a Unified Label Set](https://mlsanity.com/paper/2503.10515/) (2025)
- [Exploring the Translation Mechanism of Large Language Models](https://mlsanity.com/paper/2502.11806/) (2025)
- [Understanding Subword Compositionality of Large Language Models](https://mlsanity.com/paper/2508.17953/) (2025)
- [Debiasing Multilingual LLMs in Cross-lingual Latent Space](https://mlsanity.com/paper/2508.17948/) (2025)
- [Retrieving Implicit and Explicit Emotional Events Using Large Language Models](https://mlsanity.com/paper/2410.19128/) (2024)
- [Why Do Safety Guardrails Degrade Across Languages?](https://mlsanity.com/paper/2605.17173/) (2026)
- [Discovering Lexical Gaps Using Embeddings from Multilingual LLMs](https://mlsanity.com/paper/2605.24310/) (2026)
- [Language Shapes Instruction Hierarchy Compliance in Multilingual LLMs](https://mlsanity.com/paper/2607.23545/) (2026)

## Abstract (arXiv)

Multilingual large language models (LLMs) seem to generalize somewhat across languages. We hypothesize this is a result of implicit vector space alignment. Evaluating such alignment, we see that larger models exhibit very high-quality linear alignments between corresponding concepts in different languages. Our experiments show that multilingual LLMs suffer from two familiar weaknesses: generalization works best for languages with similar typology, and for abstract concepts. For some models, e.g., the Llama-2 family of models, prompt-based embeddings align better than word embeddings, but the projections are less linear -- an observation that holds across almost all model families, indicating that some of the implicitly learned alignments are broken somewhat by prompt-based methods.