Blessing of Multilinguality: A Systematic Analysis of Multilingual In-Context Learning
Not on the map. Why?
Abstract (arXiv)
While multilingual large language models generally perform adequately, and sometimes even rival English performance on high-resource languages (HRLs), they often significantly underperform on low-resource languages (LRLs). Among several prompting strategies aiming at bridging the gap, multilingual in-context learning (ICL) has been particularly effective when demonstration in target languages is unavailable. However, there lacks a systematic understanding of when and why it works well. In this work, we systematically analyze multilingual ICL, using demonstrations in HRLs to enhance cross-lingual transfer. We show that demonstrations in mixed HRLs consistently outperform English-only ones across the board, particularly for tasks written in LRLs. Surprisingly, our ablation study shows that the presence of irrelevant non-English sentences in the prompt yields measurable gains, suggesting the effectiveness of multilingual exposure itself. Our results highlight the potential of strategically leveraging multilingual resources to bridge the performance gap for underrepresented languages.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 47 of its 70 references that are in the map. First posted on arXiv 17 Feb 2025.
Builds on
47 of 70 references.
References · 47
- 2017Attention Is All You Needcited by 59,932
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
35 more
- 2020GLU Variants Improve Transformercited by 1,724
- 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
- 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
- 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
- 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
- 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
- 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
- 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
- 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
- 2020Beyond English-Centric Multilingual Machine Translationcited by 559
- 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
- 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
- 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
- 2021Few-shot Learning with Multilingual Language Modelscited by 278
- 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
- 2020A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarioscited by 170
- 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
- 2024How do Large Language Models Handle Multilingualism?cited by 148
- 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 121
- 2023Don't Trust ChatGPT when Your Question is not in English: A Study of Multilingual Abilities and Types of LLMscited by 103
- 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languagescited by 96
- 2021Language Models are Few-shot Multilingual Learnerscited by 92
- 2023Do Multilingual Language Models Think Better in English?cited by 87
- 2024On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neuronscited by 83
- 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
- 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
- 2021Revisiting the Primacy of English in Zero-shot Cross-lingual Transfercited by 47
- 2020XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualizationcited by 42
- 2023PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuningcited by 39
- 2024Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translationcited by 19
- 2024The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysiscited by 13
- 2024Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?cited by 11
- 2024The Roles of English in Evaluating Multilingual Language Modelscited by 7
Built on by
Citing papers per year: 2026: 3.
3 papers.