mlsanityData to 14 Oct 2026

Understanding Cross-Lingual Alignment -- A Survey

arXiv 2404.06228 ↗First posted 9 Apr 2024cs.CLcited by 34

Not on the map. Why?

Abstract (arXiv)

Cross-lingual alignment, the meaningful similarity of representations across languages in multilingual language models, has been an active field of research in recent years. We survey the literature of techniques to improve cross-lingual alignment, providing a taxonomy of methods and summarising insights from throughout the field. We present different understandings of cross-lingual alignment and their limitations. We provide a qualitative summary of results from a large number of surveyed papers. Finally, we discuss how these insights may be applied not only to encoder models, where this topic has been heavily studied, but also to encoder-decoder or even decoder-only models, and argue that an effective trade-off between language-neutral and language-specific information is key.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 83 of its 116 references that are in the map. First posted on arXiv 9 Apr 2024.

Builds on

83 of 116 references.

References · 83

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  7. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  8. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  9. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  10. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  11. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  12. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
38 more
  1. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  2. 2017Word Translation Without Parallel Datacited by 487
  3. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  4. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  5. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  6. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  7. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  8. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  9. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  10. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  11. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 255
  12. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  13. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  14. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 206
  15. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  16. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 165
  17. 2022mGPT: Few-Shot Learners Go Multilingualcited by 142
  18. 2020Multilingual Alignment of Contextual Word Representationscited by 114
  19. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
  20. 2022The Geometry of Multilingual Language Model Representationscited by 95
  21. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  22. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 87
  23. 2020On the Language Neutrality of Pre-trained Multilingual Representationscited by 85
  24. 2020CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLPcited by 79
  25. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  26. 2023Multilingual LLMs are Better Cross-lingual In-context Learners with Alignmentcited by 71
  27. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  28. 2020ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corporacited by 63
  29. 2018On the Limitations of Unsupervised Bilingual Dictionary Inductioncited by 61
  30. 2019Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmaxcited by 58
  31. 2020On Learning Universal Representations Across Languagescited by 58
  32. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  33. 2020Explicit Alignment Objectives for Multilingual Bidirectional Encoderscited by 50
  34. 2021DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoderscited by 50
  35. 2020Inducing Language-Agnostic Multilingual Representationscited by 48
  36. 2020LAReQA: Language-agnostic answer retrieval from a multilingual poolcited by 42
  37. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignmentcited by 41
  38. 2021Consistency Regularization for Cross-Lingual Fine-Tuningcited by 41

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 13; 2026: 14.

34 papers.

Sort:

By area:

  1. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 29
  2. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 28
  3. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  4. 2025Language-Specific Latent Process Hinders Cross-Lingual Performancecited by 21
  5. 2024PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignmentcited by 20
  6. 2024Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignmentcited by 14
  7. 2025Can you map it to English? The Role of Cross-Lingual Alignment in Multilingual Performance of LLMscited by 12
  8. 2025Beyond Literal Token Overlap: Token Alignability for Multilingualitycited by 9
  9. 2024Exploring the Role of Transliteration in In-Context Learning for Low-resource Languages Written in Non-Latin Scriptscited by 8
  10. 2024Contrastive Learning for Task-Independent SpeechLLM-Pretrainingcited by 8
  11. 2024How Transliterations Improve Crosslingual Alignmentcited by 6
  12. 2025The Cross-Lingual Cost: Retrieval Biases in RAG over Arabic-English Corporacited by 5
22 more
  1. 2025Conan-Embedding-v2: Training an LLM from Scratch for Text Embeddingscited by 4
  2. 2025On the Entity-Level Alignment in Crosslingual Consistencycited by 4
  3. 2025When Language Shapes Thought: Cross-Lingual Transfer of Factual Knowledge in Question Answeringcited by 3
  4. 2025Words as Bridges: Exploring Computational Support for Cross-Disciplinary Translation Workcited by 2
  5. 202520min-XD: A Comparable Corpus of Swiss News Articlescited by 2
  6. 2026A Shared Geometry of Difficulty in Multilingual Language Modelscited by 2
  7. 2025Multilingual Vision-Language Models, A Surveycited by 1
  8. 2026Towards Comprehensive Semantic Speech Embeddings for Chinese Dialectscited by 1
  9. 2026Effective QA-driven Annotation of Predicate-Argument Relations Across Languagescited by 1
  10. 2026Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretrainingcited by 1
  11. 2026DiMtextsuperscript3: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Mergingcited by 1
  12. 2026Cross-Lingual Steering for Figurative Language Generationcited by 1
  13. 2026Disentangled Contrastive Learning for Zero-Shot Multilingual Dense Retrievalcited by 1
  14. 2025Cross-Prompt Encoder for Low-Performing Languagescited by 0
  15. 2025TRepLiNa: Layer-wise CKA+REPINA Alignment Improves Low-Resource Machine Translation in Aya-23 8Bcited by 0
  16. 2026Enhancing Multilingual Embeddings via Multi-Way Parallel Text Alignmentcited by 0
  17. 2026Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoderscited by 0
  18. 2026Neither Here Nor There: Cross-Lingual Representation Dynamics of Code-Mixed Text in Multilingual Encoderscited by 0
  19. 2026Why Low-Resource NLP Needs More Than Cross-Lingual Transfer: Lessons Learned from Luxembourgishcited by 0
  20. 2026Bridging the English-Arabic Medical Knowledge Gap: Targeted Low-Rank Adaptation via Causal Layer Selectioncited by 0
  21. 2026Predicting Multilingual Classification and Translation Performance of LLMs with Cross-Lingual Alignment -- Is English Enough?cited by 0
  22. 2026There is No Theoretical Curse of Multilinguality For Embedding Space Structurecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?
  2. 2022Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment
  3. 2024Exploring Alignment in Shared Cross-lingual Spaces
  4. 2024Locally Measuring Cross-lingual Lexical Alignment: A Domain and Word Level Perspective
  5. 2025Steering into New Embedding Spaces: Analyzing Cross-Lingual Alignment Induced by Model Interventions in Multilingual Language Models
  6. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias
  7. 2026Cross-Lingual Representation Alignment by Token-Level Optimal Transport in a Language-Agnostic Space
  8. 2025AlignX: Advancing Multilingual Large Language Models with Multilingual Representation Alignment