# Improving Multilingual Instruction Finetuning via Linguistically Natural and Diverse Datasets

arXiv: https://arxiv.org/abs/2407.01853 · HTML: https://mlsanity.com/paper/2407.01853/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jul 2024.

## Built on by (most cited first, 4 of 4)

- [The Emergence of Abstract Thought in Large Language Models Beyond Any Language](https://mlsanity.com/paper/2506.09890/) (2025)
- [Lens: Rethinking Multilingual Enhancement for Large Language Models](https://mlsanity.com/paper/2410.04407/) (2024)
- [Exploring Cross-lingual Latent Transplantation: Mutual Opportunities and Open Challenges](https://mlsanity.com/paper/2412.12686/) (2024)
- [Improving Training Efficiency and Reducing Maintenance Costs via Language Specific Model Merging](https://mlsanity.com/paper/2601.16127/) (2026)

## Abstract (arXiv)

Advancements in Large Language Models (LLMs) have significantly enhanced instruction-following capabilities. However, most Instruction Fine-Tuning (IFT) datasets are predominantly in English, limiting model performance in other languages. Traditional methods for creating multilingual IFT datasets such as translating existing English IFT datasets or converting existing NLP datasets into IFT datasets by templating, struggle to capture linguistic nuances and ensure prompt (instruction) diversity. To address this issue, we propose a novel method for collecting multilingual IFT datasets that preserves linguistic naturalness and ensures prompt diversity. This approach leverages English-focused LLMs, monolingual corpora, and a scoring function to create high-quality, diversified IFT datasets in multiple languages. Experiments demonstrate that LLMs finetuned using these IFT datasets show notable improvements in both generative and discriminative tasks, indicating enhanced language comprehension by LLMs in non-English contexts. Specifically, on the multilingual summarization task, LLMs using our IFT dataset achieved 17.57% and 15.23% improvements over LLMs fine-tuned with translation-based and template-based datasets, respectively.