# Understanding Cross-Lingual Alignment -- A Survey

arXiv: https://arxiv.org/abs/2404.06228 · HTML: https://mlsanity.com/paper/2404.06228/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 83 of its 116 references that are in the map.
- First posted on arXiv 9 Apr 2024.

## Builds on (most cited first, 30 of 83)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification](https://mlsanity.com/paper/1908.11828/) (2019)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://mlsanity.com/paper/2007.07834/) (2020)
- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models](https://mlsanity.com/paper/2305.13707/) (2023)
- [Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting](https://mlsanity.com/paper/2305.07004/) (2023)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [Multilingual Alignment of Contextual Word Representations](https://mlsanity.com/paper/2002.03518/) (2020)

## Built on by (most cited first, 30 of 34)

- [Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs](https://mlsanity.com/paper/2502.14830/) (2025)
- [The Same But Different: Structural Similarities and Differences in Multilingual Language Modeling](https://mlsanity.com/paper/2410.09223/) (2024)
- [MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment](https://mlsanity.com/paper/2410.05873/) (2024)
- [Language-Specific Latent Process Hinders Cross-Lingual Performance](https://mlsanity.com/paper/2505.13141/) (2025)
- [PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignment](https://mlsanity.com/paper/2407.16222/) (2024)
- [Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment](https://mlsanity.com/paper/2406.19759/) (2024)
- [Can you map it to English? The Role of Cross-Lingual Alignment in Multilingual Performance of LLMs](https://mlsanity.com/paper/2504.09378/) (2025)
- [Beyond Literal Token Overlap: Token Alignability for Multilinguality](https://mlsanity.com/paper/2502.06468/) (2025)
- [Exploring the Role of Transliteration in In-Context Learning for Low-resource Languages Written in Non-Latin Scripts](https://mlsanity.com/paper/2407.02320/) (2024)
- [Contrastive Learning for Task-Independent SpeechLLM-Pretraining](https://mlsanity.com/paper/2412.15712/) (2024)
- [How Transliterations Improve Crosslingual Alignment](https://mlsanity.com/paper/2409.17326/) (2024)
- [The Cross-Lingual Cost: Retrieval Biases in RAG over Arabic-English Corpora](https://mlsanity.com/paper/2507.07543/) (2025)
- [Conan-Embedding-v2: Training an LLM from Scratch for Text Embeddings](https://mlsanity.com/paper/2509.12892/) (2025)
- [On the Entity-Level Alignment in Crosslingual Consistency](https://mlsanity.com/paper/2510.10280/) (2025)
- [When Language Shapes Thought: Cross-Lingual Transfer of Factual Knowledge in Question Answering](https://mlsanity.com/paper/2505.24409/) (2025)
- [Words as Bridges: Exploring Computational Support for Cross-Disciplinary Translation Work](https://mlsanity.com/paper/2503.18471/) (2025)
- [20min-XD: A Comparable Corpus of Swiss News Articles](https://mlsanity.com/paper/2504.21677/) (2025)
- [A Shared Geometry of Difficulty in Multilingual Language Models](https://mlsanity.com/paper/2601.12731/) (2026)
- [Multilingual Vision-Language Models, A Survey](https://mlsanity.com/paper/2509.22123/) (2025)
- [Towards Comprehensive Semantic Speech Embeddings for Chinese Dialects](https://mlsanity.com/paper/2601.07274/) (2026)
- [Effective QA-driven Annotation of Predicate-Argument Relations Across Languages](https://mlsanity.com/paper/2602.22865/) (2026)
- [Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining](https://mlsanity.com/paper/2604.17633/) (2026)
- [DiMtextsuperscript3: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Merging](https://mlsanity.com/paper/2605.12960/) (2026)
- [Cross-Lingual Steering for Figurative Language Generation](https://mlsanity.com/paper/2605.30443/) (2026)
- [Disentangled Contrastive Learning for Zero-Shot Multilingual Dense Retrieval](https://mlsanity.com/paper/2608.02189/) (2026)
- [Cross-Prompt Encoder for Low-Performing Languages](https://mlsanity.com/paper/2508.10352/) (2025)
- [TRepLiNa: Layer-wise CKA+REPINA Alignment Improves Low-Resource Machine Translation in Aya-23 8B](https://mlsanity.com/paper/2510.06249/) (2025)
- [Enhancing Multilingual Embeddings via Multi-Way Parallel Text Alignment](https://mlsanity.com/paper/2602.21543/) (2026)
- [Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoders](https://mlsanity.com/paper/2603.18863/) (2026)
- [Neither Here Nor There: Cross-Lingual Representation Dynamics of Code-Mixed Text in Multilingual Encoders](https://mlsanity.com/paper/2603.19771/) (2026)

## Abstract (arXiv)

Cross-lingual alignment, the meaningful similarity of representations across languages in multilingual language models, has been an active field of research in recent years. We survey the literature of techniques to improve cross-lingual alignment, providing a taxonomy of methods and summarising insights from throughout the field. We present different understandings of cross-lingual alignment and their limitations. We provide a qualitative summary of results from a large number of surveyed papers. Finally, we discuss how these insights may be applied not only to encoder models, where this topic has been heavily studied, but also to encoder-decoder or even decoder-only models, and argue that an effective trade-off between language-neutral and language-specific information is key.