mlsanityData to 13 Oct 2026

Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment

arXiv 2311.08089 ↗First posted 14 Nov 2023cs.CLcited by 33

Not on the map. Why?

Abstract (arXiv)

Multilingual generative models obtain remarkable cross-lingual in-context learning capabilities through pre-training on large-scale corpora. However, they still exhibit a performance bias toward high-resource languages and learn isolated distributions of multilingual sentence representations, which may hinder knowledge transfer across languages. To bridge this gap, we propose a simple yet effective cross-lingual alignment framework exploiting pairs of translation sentences. It aligns the internal sentence representations across different languages via multilingual contrastive learning and aligns outputs by following cross-lingual instructions in the target language. Experimental results show that even with less than 0.1 textperthousand of pre-training tokens, our alignment framework significantly boosts the cross-lingual abilities of generative language models and mitigates the performance gap. Further analyses reveal that it results in a better internal multilingual representation distribution of multilingual models.

Key facts. Cited by 33 papers in the map (data through 13 Oct 2026). Builds on 43 of its 57 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

43 of 57 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,106
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,107
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,884
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,765
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,693
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,170
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,978
  11. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,815
  12. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,397
31 more
  1. 2017Mixed Precision Trainingcited by 1,147
  2. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,033
  3. 2020COMET: A Neural Framework for MT Evaluationcited by 1,027
  4. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 914
  5. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 600
  6. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  7. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 555
  8. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  9. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 332
  10. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 304
  11. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  12. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  13. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 263
  14. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  15. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  16. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  17. 2021Understanding the Capabilities, Limitations, and Societal Impact of Large Language Modelscited by 159
  18. 2020Multilingual Alignment of Contextual Word Representationscited by 114
  19. 2021Contrastive Learning for Many-to-many Multilingual Neural Machine Translationcited by 108
  20. 2023Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptationcited by 80
  21. 2023Multilingual LLMs are Better Cross-lingual In-context Learners with Alignmentcited by 71
  22. 2021It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoningcited by 69
  23. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  24. 2023PolyLM: An Open Source Polyglot Large Language Modelcited by 64
  25. 2020On Learning Universal Representations Across Languagescited by 58
  26. 2022AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Modelcited by 57
  27. 2021Improved Text Classification via Contrastive Adversarial Trainingcited by 38
  28. 2023Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrationscited by 34
  29. 2020Multilingual BERT Post-Pretraining Alignmentcited by 30
  30. 2021Aligning Cross-lingual Sentence Representations with Dual Momentum Contrastcited by 7
  31. 2023Optimal Transport Posterior Alignment for Cross-lingual Semantic Parsingcited by 2

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 7; 2025: 17; 2026: 9.

33 papers.

Sort:

By area:

  1. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  2. 2025Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Studycited by 38
  3. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  4. 2025A Survey of Multilingual Reasoning in Language Modelscited by 32
  5. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  6. 2025MPO: Multilingual Safety Alignment via Reward Gap Optimizationcited by 18
  7. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
  8. 2025AdaMCoT: Rethinking Cross-Lingual Factual Reasoning through Adaptive Multilingual Chain-of-Thoughtcited by 14
  9. 2025Enhancing Non-English Capabilities of English-Centric Large Language Models through Deep Supervision Fine-Tuningcited by 11
  10. 2025Can you map it to English? The Role of Cross-Lingual Alignment in Multilingual Performance of LLMscited by 11
  11. 2025TokAlign: Efficient Vocabulary Adaptation via Token Alignmentcited by 11
  12. 2025Steering into New Embedding Spaces: Analyzing Cross-Lingual Alignment Induced by Model Interventions in Multilingual Language Modelscited by 10
21 more
  1. 2025Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignmentcited by 9
  2. 2025The Translation Barrier Hypothesis: Multilingual Generation with Large Language Models Suffers from Implicit Translation Failurecited by 9
  3. 2026Align Once, Benefit Multilingually: Enforcing Multilingual Consistency for LLM Safety Alignmentcited by 9
  4. 2024Facilitating large language model Russian adaptation with Learned Embedding Propagationcited by 8
  5. 2025Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineeringcited by 8
  6. 2025AlignX: Advancing Multilingual Large Language Models with Multilingual Representation Alignmentcited by 6
  7. 2024Lens: Rethinking Multilingual Enhancement for Large Language Modelscited by 5
  8. 2025When Language Shapes Thought: Cross-Lingual Transfer of Factual Knowledge in Question Answeringcited by 3
  9. 2025Gradual Code-Switching as Inference-Time Cross-Lingual Representational Alignment for LLMscited by 3
  10. 2026Language Steering for Multilingual In-Context Learningcited by 3
  11. 2026LASA: Language-Agnostic Semantic Alignment at the Semantic Bottleneck for LLM Safetycited by 3
  12. 2025CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignmentcited by 2
  13. 2025Hierarchical Level-Wise News Article Clustering via Multilingual Matryoshka Embeddingscited by 2
  14. 2025A Survey of Large Language Models in Discipline-specific Research: Challenges, Methods and Opportunitiescited by 2
  15. 2026Building a Strong Instruction Language Model for a Less-Resourced Languagecited by 1
  16. 2024Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Modelcited by 0
  17. 2026Multilinguality as Sense Adaptationcited by 0
  18. 2026Meta-Learning Preferences for Multilingual LLM Alignmentcited by 0
  19. 2026Language Chain in Alignment: Cross-lingual Ranking Preference Optimizationcited by 0
  20. 2026Cross-Lingual Representation Alignment by Token-Level Optimal Transport in a Language-Agnostic Spacecited by 0
  21. 2026Cross-Lingual Alignment for Decoder-Only Models using MoE Routerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Align, Generate, Learn: A Novel Closed-Loop Framework for Cross-Lingual In-Context Learning
  2. 2026Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?
  3. 2022Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment
  4. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs
  5. 2024An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inference
  6. 2023Exploring the Relationship between Alignment and Cross-lingual Transfer in Multilingual Transformers
  7. 2024Exploring Alignment in Shared Cross-lingual Spaces
  8. 2024CrossIn: An Efficient Instruction Tuning Approach for Cross-Lingual Knowledge Alignment