mlsanityData to 14 Oct 2026

Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca

arXiv 2309.08958 ↗First posted 16 Sep 2023cs.CL cs.AIcited by 60

Not on the map. Why?

Abstract (arXiv)

Foundational large language models (LLMs) can be instruction-tuned to perform open-domain question answering, facilitating applications like chat assistants. While such efforts are often carried out in a single language, we empirically analyze cost-efficient strategies for multilingual scenarios. Our study employs the Alpaca dataset and machine translations of it to form multilingual data, which is then used to tune LLMs through either low-rank adaptation or full-parameter training. Under a controlled computation budget, comparisons show that multilingual tuning is on par or better than tuning a model for each language. Furthermore, multilingual tuning with downsampled data can be as powerful and more robust. Our findings serve as a guide for expanding language support through instruction tuning.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Builds on 34 of its 56 references that are in the map. First posted on arXiv 16 Sep 2023.

Builds on

34 of 56 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  12. 2018Deep contextualized word representationscited by 2,290
22 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  5. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  6. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  7. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  8. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  9. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  10. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  11. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
  12. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  13. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  14. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 92
  15. 2023Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptationcited by 81
  16. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  17. 2023PolyLM: An Open Source Polyglot Large Language Modelcited by 64
  18. 2023Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capabilitycited by 62
  19. 2023Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructionscited by 56
  20. 2023An Open Dataset and Model for Language Identificationcited by 50
  21. 2023Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrationscited by 35
  22. 2023Language Versatilists vs. Specialists: An Empirical Revisiting on Multilingual Transfer Abilitycited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 28; 2025: 17; 2026: 12.

60 papers.

Sort:

By area:

  1. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  2. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 177
  3. 2024Large Language Models Meet NLP: A Surveycited by 103
  4. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
  5. 2024Multilingual Instruction Tuning With Just a Pinch of Multilingualitycited by 73
  6. 2024Adapting Large Language Models for Document-Level Machine Translationcited by 59
  7. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  8. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedbackcited by 51
  9. 2023PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuningcited by 39
  10. 2023Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?cited by 39
  11. 2024LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMscited by 33
  12. 2024MaLA-500: Massive Language Adaptation of Large Language Modelscited by 28
38 more
  1. 2025Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoningcited by 24
  2. 2024Fine-tuning Large Language Models with Sequential Instructionscited by 20
  3. 2024Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice?cited by 17
  4. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  5. 2024How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs?cited by 13
  6. 2024Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translationcited by 13
  7. 2024Zero-shot cross-lingual transfer in instruction tuning of large language modelscited by 11
  8. 2025Improving Multilingual Language Models by Aligning Representations through Steeringcited by 11
  9. 2025Cross-lingual Collapse: How Language-Centric Foundation Models Shape Reasoning in Large Language Modelscited by 11
  10. 2025Blackbox Model Provenance via Palimpsestic Membership Inferencecited by 11
  11. 2025SinLlama -- A Large Language Model for Sinhalacited by 9
  12. 2024Accelerating Multilingual Language Model for Excessively Tokenized Languagescited by 8
  13. 2024Exploring the Role of Transliteration in In-Context Learning for Low-resource Languages Written in Non-Latin Scriptscited by 8
  14. 2024M2Lingual: Enhancing Multilingual, Multi-Turn Instruction Alignment in Large Language Modelscited by 7
  15. 2025DocHPLT: A Massively Multilingual Document-Level Translation Datasetcited by 7
  16. 2024On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruningcited by 6
  17. 2025A Post-trainer's Guide to Multilingual Training Data: Uncovering Cross-lingual Transfer Dynamicscited by 6
  18. 2024Can Code-Switched Texts Activate a Knowledge Switch in LLMs? A Case Study on English-Korean Code-Switchingcited by 5
  19. 2026Cross-Lingual Activation Steering for Multilingual Language Modelscited by 5
  20. 2024Is It Good Data for Multilingual Instruction Tuning or Just Bad Multilingual Evaluation for Large Language Models?cited by 4
  21. 2025UPDESH: Synthesizing Grounded Instruction Tuning Data for 13 Indic Languagescited by 4
  22. 2023The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language Heuristicscited by 3
  23. 2024Fine-tuning with HED-IT: The impact of human post-editing for dialogical language modelscited by 3
  24. 2025Got Compute, but No Data: Lessons From Post-training a Finnish LLMcited by 3
  25. 2025CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignmentcited by 3
  26. 2026Preferences for Idiomatic Language are Acquired Slowly -- and Forgotten Quickly: A Case Study on Swedishcited by 3
  27. 2024Train More Parameters But Mind Their Placement: Insights into Language Adaptation with PEFTcited by 2
  28. 2024Exploring Cross-lingual Latent Transplantation: Mutual Opportunities and Open Challengescited by 2
  29. 2025HBO: Hierarchical Balancing Optimization for Fine-Tuning Large Language Modelscited by 2
  30. 2026One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Mergingcited by 2
  31. 2026Parallel-SFT: Improving Zero-Shot Cross-Programming-Language Transfer for Code RLcited by 2
  32. 2024Hallucinations or Attention Misdirection? The Path to Strategic Value Extraction in Business Using Large Language Modelscited by 1
  33. 2024Evaluation of Few-Shot Learning for Classification Tasks in the Polish Languagecited by 1
  34. 2024Bridging the Language Gap: Enhancing Multilingual Prompt-Based Code Generation in LLMs via Zero-Shot Cross-Lingual Transfercited by 1
  35. 2024No Optimal Language Set Exists for Multilingual Instruction Tuning: Insights from a Linguistically-Informed Studycited by 1
  36. 2025XL-Suite: Cross-Lingual Synthetic Training and Evaluation Data for Open-Ended Generationcited by 1
  37. 2025Building Instruction-Tuning Datasets from Human-Written Instructions with Open-Weight Large Language Modelscited by 1
  38. 2025CC-Tuning: A Cross-Lingual Connection Mechanism for Improving Joint Multilingual Supervised Fine-Tuningcited by 1

50 most cited of 60 All 60 (JSON)