mlsanityData to 7 Oct 2026

Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?

arXiv 2312.12683 ↗First posted 20 Dec 2023cs.CLcited by 39

Not on the map. Why?

Abstract (arXiv)

The vast majority of today's large language models (LLMs) are English-centric, having been pretrained predominantly on English text. Yet, in order to meet user expectations, models need to be able to respond appropriately in multiple languages once deployed in downstream applications. This requires strong cross-lingual transfer abilities. In this work, we investigate the minimal amount of multilinguality required during finetuning to elicit cross-lingual generalisation in English-centric LLMs. In experiments across four LLMs, we find that multilingual instruction tuning with as few as two to three languages is both necessary and sufficient to elicit effective cross-lingual generalisation, with the limiting factor being the degree to which a target language is seen during pretraining. Evaluations on five different tasks further reveal that multilingual instruction tuning is most beneficial for generative tasks that assume input/output language agreement, such as in chat settings, while being of less importance for highly structured classification-style tasks. Our code and data is available at https://github.com/ZurichNLP/multilingual-instruction-tuning.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 52 of its 85 references that are in the map. First posted on arXiv 20 Dec 2023.

Builds on

52 of 85 references.

References · 52

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  12. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
38 more
  1. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  3. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  4. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  5. 2023LIMA: Less Is More for Alignmentcited by 1,080
  6. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  7. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  8. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  9. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  10. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  11. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  12. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  13. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  14. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  15. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  16. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  17. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  18. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  19. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  20. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  21. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  22. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  23. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  24. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  25. 2020Neural CRF Model for Sentence Alignment in Text Simplificationcited by 106
  26. 2020ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformationscited by 101
  27. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  28. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 90
  29. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 87
  30. 2021Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoningcited by 64
  31. 2023Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpacacited by 60
  32. 2023Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOMcited by 58
  33. 2021A Systematic Investigation of Commonsense Knowledge in Large Language Modelscited by 56
  34. 2023An Open Dataset and Model for Language Identificationcited by 49
  35. 2023PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuningcited by 39
  36. 2023BLESS: Benchmarking Large Language Models on Sentence Simplificationcited by 34
  37. 2023Revisiting non-English Text Simplification: A Unified Multilingual Benchmarkcited by 27
  38. 2022Causes and Cures for Interference in Multilingual Translationcited by 23

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 7; 2026: 5.

39 papers.

Sort:

By area:

  1. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
  2. 2024Multilingual Instruction Tuning With Just a Pinch of Multilingualitycited by 71
  3. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  4. 2024Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languagescited by 36
  5. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  6. 2024LangBridge: Multilingual Reasoning Without Multilingual Supervisioncited by 22
  7. 2024Poro 34B and the Blessing of Multilingualitycited by 18
  8. 2024Self-Alignment: Improving Alignment of Cultural Values in LLMs via In-Context Learningcited by 18
  9. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  10. 2024Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice?cited by 16
  11. 2024Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settingscited by 14
  12. 2024How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs?cited by 13
27 more
  1. 2024Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translationcited by 13
  2. 2024Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?cited by 11
  3. 2024Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learnerscited by 11
  4. 2025Cross-lingual Collapse: How Language-Centric Foundation Models Shape Reasoning in Large Language Modelscited by 11
  5. 2024Zero-shot cross-lingual transfer in instruction tuning of large language modelscited by 10
  6. 2024Aurora-M: Open Source Continual Pre-training for Multilingual Language and Codecited by 10
  7. 2024CrossIn: An Efficient Instruction Tuning Approach for Cross-Lingual Knowledge Alignmentcited by 10
  8. 2024mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humanscited by 10
  9. 2025Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Modelscited by 8
  10. 2024The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insightscited by 7
  11. 2024The Impact of Language Adapters in Cross-Lingual Transfer for NLUcited by 6
  12. 2024Multilingual Arbitrage: Optimizing Data Pools to Accelerate Multilingual Progresscited by 6
  13. 2025A Post-trainer's Guide to Multilingual Training Data: Uncovering Cross-lingual Transfer Dynamicscited by 6
  14. 2024Analysis of Multi-Source Language Training in Cross-Lingual Transfercited by 5
  15. 2024Is It Good Data for Multilingual Instruction Tuning or Just Bad Multilingual Evaluation for Large Language Models?cited by 4
  16. 2025Understanding LLMs' Cross-Lingual Context Retrieval: How Good It Is And Where It Comes Fromcited by 4
  17. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Modelcited by 3
  18. 2026LinguaMap: Which Layers of LLMs Speak Your Language and How to Tune Them?cited by 3
  19. 2024The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Modelcited by 2
  20. 2024Train More Parameters But Mind Their Placement: Insights into Language Adaptation with PEFTcited by 2
  21. 2024Exploring Cross-lingual Latent Transplantation: Mutual Opportunities and Open Challengescited by 1
  22. 2025HBO: Hierarchical Balancing Optimization for Fine-Tuning Large Language Modelscited by 1
  23. 2025The Effect of Language Diversity When Fine-Tuning Large Language Models for Translationcited by 1
  24. 2026English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Trainingcited by 0
  25. 2026Left-Branching Transformers Excel at Right-Branching Languages: Data Shapes Word Order Preferences in Language Modelscited by 0
  26. 2026Language Chain in Alignment: Cross-lingual Ranking Preference Optimizationcited by 0
  27. 2026Assessing the Impact of Language Disparity on Multilingual Linguistic Ability in Large Language Modelscited by 0