mlsanityData to 14 Oct 2026

Multilingual Instruction Tuning With Just a Pinch of Multilinguality

arXiv 2401.01854 ↗First posted 3 Jan 2024cs.CL cs.AI cs.LGcited by 73

Not on the map. Why?

Abstract (arXiv)

As instruction-tuned large language models (LLMs) gain global adoption, their ability to follow instructions in multiple languages becomes increasingly crucial. In this work, we investigate how multilinguality during instruction tuning of a multilingual LLM affects instruction-following across languages from the pre-training corpus. We first show that many languages transfer some instruction-following capabilities to other languages from even monolingual tuning. Furthermore, we find that only 40 multilingual examples integrated in an English tuning set substantially improve multilingual instruction-following, both in seen and unseen languages during tuning. In general, we observe that models tuned on multilingual mixtures exhibit comparable or superior performance in multiple languages compared to monolingually tuned models, despite training on 10x fewer examples in those languages. Finally, we find that diversifying the instruction tuning set with even just 2-4 languages significantly improves cross-lingual generalization. Our results suggest that building massively multilingual instruction-tuned models can be done with only a very small set of multilingual instruction-responses.

Key facts. Cited by 73 papers in the map (data through 14 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 3 Jan 2024.

Builds on

32 of 50 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
20 more
  1. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  2. 2023LIMA: Less Is More for Alignmentcited by 1,100
  3. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  4. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  5. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  6. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  7. 2019How multilingual is Multilingual BERT?cited by 645
  8. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  9. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  10. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  11. 2023The False Promise of Imitating Proprietary LLMscited by 263
  12. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  13. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  14. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 181
  15. 2023Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpacacited by 60
  16. 2020English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Toocited by 39
  17. 2023PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuningcited by 39
  18. 2023Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?cited by 39
  19. 2022A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bankcited by 29
  20. 2022Causes and Cures for Interference in Multilingual Translationcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 26; 2025: 35; 2026: 12.

73 papers.

Sort:

By area:

  1. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  2. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  3. 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformerscited by 44
  4. 2024Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languagescited by 36
  5. 2025ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfercited by 29
  6. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignmentcited by 26
  7. 2025Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoningcited by 24
  8. 2024Unveiling Linguistic Regions in Large Language Modelscited by 22
  9. 2025Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMscited by 22
  10. 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 22
  11. 2024Unlocking the Potential of Model Merging for Low-Resource Languagescited by 19
  12. 2024HyperCLOVA X Technical Reportcited by 18
38 more
  1. 2024Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challengescited by 18
  2. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  3. 2024Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translationcited by 13
  4. 2024MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructionscited by 12
  5. 2025Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basquecited by 12
  6. 2024Zero-shot cross-lingual transfer in instruction tuning of large language modelscited by 11
  7. 2024Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?cited by 11
  8. 2024CrossIn: An Efficient Instruction Tuning Approach for Cross-Lingual Knowledge Alignmentcited by 11
  9. 2025XIFBench: Evaluating Large Language Models on Multilingual Instruction Followingcited by 11
  10. 2025Cross-lingual Collapse: How Language-Centric Foundation Models Shape Reasoning in Large Language Modelscited by 11
  11. 2025Aligning LLMs for Multilingual Consistency in Enterprise Applicationscited by 11
  12. 2024Aurora-M: Open Source Continual Pre-training for Multilingual Language and Codecited by 10
  13. 2024Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models?cited by 10
  14. 2024Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Modelscited by 9
  15. 2025Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Modelscited by 8
  16. 2025Can Prompting LLMs Unlock Hate Speech Detection across Languages? A Zero-shot and Few-shot Studycited by 7
  17. 2025A Post-trainer's Guide to Multilingual Training Data: Uncovering Cross-lingual Transfer Dynamicscited by 6
  18. 2025It's All About In-Context Learning! Teaching Extremely Low-Resource Languages to LLMscited by 6
  19. 2024Analysis of Multi-Source Language Training in Cross-Lingual Transfercited by 5
  20. 2026Cross-Lingual Activation Steering for Multilingual Language Modelscited by 5
  21. 2025CoCo-CoLa: Evaluating and Improving Language Adherence in Multilingual LLMscited by 4
  22. 2025Understanding LLMs' Cross-Lingual Context Retrieval: How Good It Is And Where It Comes Fromcited by 4
  23. 2025SenWiCh: Sense-Annotation of Low-Resource Languages for WiC using Hybrid Methodscited by 4
  24. 2025IMPACT: Inflectional Morphology Probes Across Complex Typologiescited by 4
  25. 2025UPDESH: Synthesizing Grounded Instruction Tuning Data for 13 Indic Languagescited by 4
  26. 2024MAPLE: Multilingual Evaluation of Parameter Efficient Finetuning of Large Language Modelscited by 3
  27. 2024Mitigating Multilingual Hallucination in Large Vision-Language Modelscited by 3
  28. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Modelcited by 3
  29. 2025Got Compute, but No Data: Lessons From Post-training a Finnish LLMcited by 3
  30. 2025CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignmentcited by 3
  31. 2025Kuwain 1.5B: An Arabic SLM via Language Injectioncited by 3
  32. 2025Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluationcited by 3
  33. 2025HBO: Hierarchical Balancing Optimization for Fine-Tuning Large Language Modelscited by 2
  34. 2025Cross-Lingual Optimization for Language Transfer in Large Language Modelscited by 2
  35. 2025Do Large Language Models Know Folktales? A Case Study of Yokai in Japanese Folktalescited by 2
  36. 2025LiveCLKTBench: Towards Reliable Evaluation of Cross-Lingual Knowledge Transfer in Multilingual LLMscited by 2
  37. 2026Parallel-SFT: Improving Zero-Shot Cross-Programming-Language Transfer for Code RLcited by 2
  38. 2024DIALIGHT: Lightweight Multilingual Development and Evaluation of Task-Oriented Dialogue Systems with Large Language Modelscited by 1

50 most cited of 73 All 73 (JSON)