# Exploring Alignment in Shared Cross-lingual Spaces

arXiv: https://arxiv.org/abs/2405.14535 · HTML: https://mlsanity.com/paper/2405.14535/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 29 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [Multilingual Alignment of Contextual Word Representations](https://mlsanity.com/paper/2002.03518/) (2020)
- [What Happens To BERT Embeddings During Fine-tuning?](https://mlsanity.com/paper/2004.14448/) (2020)
- [Discovering Latent Concepts Learned in BERT](https://mlsanity.com/paper/2205.07237/) (2022)
- [Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERT](https://mlsanity.com/paper/2101.11043/) (2021)
- [Analyzing Encoded Concepts in Transformer Language Models](https://mlsanity.com/paper/2206.13289/) (2022)
- [Asking without Telling: Exploring Latent Ontologies in Contextual Representations](https://mlsanity.com/paper/2004.14513/) (2020)
- [How transfer learning impacts linguistic knowledge in deep NLP models?](https://mlsanity.com/paper/2105.15179/) (2021)
- [NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networks](https://mlsanity.com/paper/1812.09359/) (2018)
- [Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings](https://mlsanity.com/paper/2311.18034/) (2023)
- [Can LLMs facilitate interpretation of pre-trained language models?](https://mlsanity.com/paper/2305.13386/) (2023)
- [NeuroX Library for Neuron Analysis of Deep NLP Models](https://mlsanity.com/paper/2305.17073/) (2023)
- [Scaling up Discovery of Latent Concepts in Deep NLP Models](https://mlsanity.com/paper/2308.10263/) (2023)
- [Latent Topology Induction for Understanding Contextualized Representations](https://mlsanity.com/paper/2206.01512/) (2022)
- [Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalization](https://mlsanity.com/paper/2310.12794/) (2023)

## Built on by (most cited first, 13 of 13)

- [Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers](https://mlsanity.com/paper/2411.08745/) (2024)
- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignment](https://mlsanity.com/paper/2507.14900/) (2025)
- [How Does Alignment Enhance LLMs' Multilingual Capabilities? A Language Neurons Perspective](https://mlsanity.com/paper/2505.21505/) (2025)
- [Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent?](https://mlsanity.com/paper/2507.12838/) (2025)
- [Language Surgery in Multilingual Large Language Models](https://mlsanity.com/paper/2506.12450/) (2025)
- [Beyond the Rosetta Stone: Unification Forces in Generalization Dynamics](https://mlsanity.com/paper/2508.11017/) (2025)
- [Editing Across Languages: A Survey of Multilingual Knowledge Editing](https://mlsanity.com/paper/2505.14393/) (2025)
- [Steering the Language Axis: From Linear Decodability to Causal Control](https://mlsanity.com/paper/2608.12334/) (2026)
- [An Exploration of Knowledge Editing for Arabic](https://mlsanity.com/paper/2507.09629/) (2025)
- [Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoders](https://mlsanity.com/paper/2603.18863/) (2026)
- [Harnessing Linguistic Dissimilarity for Language Generalization on Unseen Low-Resource Varieties](https://mlsanity.com/paper/2605.04500/) (2026)
- [There is No Theoretical Curse of Multilinguality For Embedding Space Structure](https://mlsanity.com/paper/2608.17088/) (2026)

## Abstract (arXiv)

Despite their remarkable ability to capture linguistic nuances across diverse languages, questions persist regarding the degree of alignment between languages in multilingual embeddings. Drawing inspiration from research on high-dimensional representations in neural language models, we employ clustering to uncover latent concepts within multilingual models. Our analysis focuses on quantifying the textitalignment and textitoverlap of these concepts across various languages within the latent space. To this end, we introduce two metrics CA and CO aimed at quantifying these aspects, enabling a deeper exploration of multilingual embeddings. Our study encompasses three multilingual models (textttmT5, textttmBERT, and textttXLM-R) and three downstream tasks (Machine Translation, Named Entity Recognition, and Sentiment Analysis). Key findings from our analysis include: i) deeper layers in the network demonstrate increased cross-lingual textitalignment due to the presence of language-agnostic concepts, ii) fine-tuning of the models enhances textitalignment within the latent space, and iii) such task-specific calibration helps in explaining the emergence of zero-shot capabilities in the models.footnoteThe code is available at urlhttps://github.com/baselmousi/multilingual-latent-concepts