# Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions?

arXiv: https://arxiv.org/abs/2402.13703 · HTML: https://mlsanity.com/paper/2402.13703/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 21 Feb 2024.

## Builds on (most cited first, 23 of 23)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets](https://mlsanity.com/paper/2307.10928/) (2023)
- [Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.16039/) (2023)
- [Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning](https://mlsanity.com/paper/2402.06619/) (2024)
- [Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine](https://mlsanity.com/paper/2301.08745/) (2023)
- [Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation](https://mlsanity.com/paper/2305.15011/) (2023)
- [It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning](https://mlsanity.com/paper/2106.12066/) (2021)
- [A Few More Examples May Be Worth Billions of Parameters](https://mlsanity.com/paper/2110.04374/) (2021)
- [PHOENIX: Open-Source Language Adaption for Direct Preference Optimization](https://mlsanity.com/paper/2401.10580/) (2024)

## Built on by (most cited first, 10 of 10)

- [Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs](https://mlsanity.com/paper/2410.03730/) (2024)
- [ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilinguality](https://mlsanity.com/paper/2510.22037/) (2025)
- [Multilingual Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2411.11072/) (2024)
- [XIFBench: Evaluating Large Language Models on Multilingual Instruction Following](https://mlsanity.com/paper/2503.07539/) (2025)
- [No Optimal Language Set Exists for Multilingual Instruction Tuning: Insights from a Linguistically-Informed Study](https://mlsanity.com/paper/2410.07809/) (2024)
- [GemMaroc: Unlocking Darija Proficiency in LLMs with Minimal Data](https://mlsanity.com/paper/2505.17082/) (2025)
- [LuxInstruct: A Cross-Lingual Instruction Tuning Dataset For Luxembourgish](https://mlsanity.com/paper/2510.07074/) (2025)
- [KITE: A Benchmark for Evaluating Korean Instruction-Following Abilities in Large Language Models](https://mlsanity.com/paper/2510.15558/) (2025)
- [LuxIT: A Luxembourgish Instruction Tuning Dataset from Monolingual Seed Data](https://mlsanity.com/paper/2510.24434/) (2025)
- [MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages](https://mlsanity.com/paper/2607.00890/) (2026)

## Abstract (arXiv)

The adaption of multilingual pre-trained LLMs into eloquent and helpful assistants is essential to facilitate their use across different language regions. In that spirit, we are the first to conduct an extensive study of the performance of multilingual models instruction-tuned on different language compositions on parallel instruction-tuning benchmarks across a selection of the most spoken Indo-European languages. We systematically examine the effects of language and instruction dataset size on a mid-sized and a large, multilingual LLMs by instruction-tuning them on parallel instruction-tuning datasets. Our results demonstrate that instruction-tuning on parallel instead of monolingual corpora benefits cross-lingual instruction following capabilities by up to 9.9%. Furthermore, we show that the Superficial Alignment Hypothesis does not hold in general, as the investigated multilingual 7B parameter model presents a counter-example requiring large-scale instruction-tuning datasets. Finally, we conduct a human annotation study to understand the alignment between human-based and GPT-4-based evaluation within multilingual chat scenarios.