mlsanityData to 14 Oct 2026

Concept Space Alignment in Multilingual LLMs

arXiv 2410.01079 ↗First posted 1 Oct 2024cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Multilingual large language models (LLMs) seem to generalize somewhat across languages. We hypothesize this is a result of implicit vector space alignment. Evaluating such alignment, we see that larger models exhibit very high-quality linear alignments between corresponding concepts in different languages. Our experiments show that multilingual LLMs suffer from two familiar weaknesses: generalization works best for languages with similar typology, and for abstract concepts. For some models, e.g., the Llama-2 family of models, prompt-based embeddings align better than word embeddings, but the projections are less linear -- an observation that holds across almost all model families, indicating that some of the implicitly learned alignments are broken somewhat by prompt-based methods.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 23 of its 40 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

23 of 40 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  5. 2019How multilingual is Multilingual BERT?cited by 645
  6. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  7. 2017Word Translation Without Parallel Datacited by 487
  8. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  9. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  10. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  11. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 118
  12. 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 86
11 more
  1. 2018On the Limitations of Unsupervised Bilingual Dictionary Inductioncited by 61
  2. 2020Explicit Alignment Objectives for Multilingual Bidirectional Encoderscited by 50
  3. 2024xCoT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoningcited by 46
  4. 2020Multilingual BERT Post-Pretraining Alignmentcited by 30
  5. 2024PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignmentcited by 20
  6. 2023Language Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models?cited by 19
  7. 2022Do language models have coherent mental models of everyday things?cited by 13
  8. 2019Lost in Evaluation: Misleading Benchmarks for Bilingual Dictionary Inductioncited by 10
  9. 2018A Discriminative Latent-Variable Model for Bilingual Lexicon Inductioncited by 8
  10. 2024On the Tip of the Tongue: Analyzing Conceptual Representation in Large Language Models with Reverse-Dictionary Probecited by 6
  11. 2017Limitations of Cross-Lingual Learning from Image Searchcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 5.

13 papers.

Sort:

By area:

  1. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  2. 2025Hypothesis-Driven Feature Manifold Analysis in LLMs via Supervised Multi-Dimensional Scalingcited by 8
  3. 2026Differential syntactic and semantic encoding in LLMscited by 7
  4. 2026When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Trainingcited by 5
  5. 2024Tomato, Tomahto, Tomate: Do Multilingual Language Models Understand Based on Subword-Level Semantic Concepts?cited by 4
  6. 2025Probing LLMs for Multilingual Discourse Generalization Through a Unified Label Setcited by 4
  7. 2025Exploring the Translation Mechanism of Large Language Modelscited by 3
  8. 2025Understanding Subword Compositionality of Large Language Modelscited by 3
  9. 2025Debiasing Multilingual LLMs in Cross-lingual Latent Spacecited by 2
  10. 2024Retrieving Implicit and Explicit Emotional Events Using Large Language Modelscited by 0
  11. 2026Why Do Safety Guardrails Degrade Across Languages?cited by 0
  12. 2026Discovering Lexical Gaps Using Embeddings from Multilingual LLMscited by 0
1 more
  1. 2026Language Shapes Instruction Hierarchy Compliance in Multilingual LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias
  2. 2024Exploring Alignment in Shared Cross-lingual Spaces
  3. 2025AlignX: Advancing Multilingual Large Language Models with Multilingual Representation Alignment
  4. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs
  5. 2026Align Once, Benefit Multilingually: Enforcing Multilingual Consistency for LLM Safety Alignment
  6. 2025How Language Directions Align with Token Geometry in Multilingual LLMs
  7. 2024Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners
  8. 2025Steering into New Embedding Spaces: Analyzing Cross-Lingual Alignment Induced by Model Interventions in Multilingual Language Models