mlsanityData to 14 Oct 2026

MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions

arXiv 2409.12958 ↗First posted 19 Sep 2024cs.CL cs.AI cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Instruction tuning enhances large language models (LLMs) by aligning them with human preferences across diverse tasks. Traditional approaches to create instruction tuning datasets face serious challenges for low-resource languages due to their dependence on data annotation. This work introduces a novel method, Multilingual Reverse Instructions (MURI), which generates high-quality instruction tuning datasets for low-resource languages without requiring human annotators or pre-existing multilingual models. Utilizing reverse instructions and a translation pipeline, MURI produces instruction-output pairs from existing human-written texts in low-resource languages. This method ensures cultural relevance and diversity by sourcing texts from different native domains and applying filters to eliminate inappropriate content. Our dataset, MURI-IT, includes more than 2 million instruction-output pairs across 200 languages. Evaluation by native speakers and fine-tuning experiments with mT5 models demonstrate the approach's effectiveness for both NLU and open-ended generation. We publicly release datasets and models at https://github.com/akoksal/muri.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 51 of its 87 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

51 of 87 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  8. 2023Mistral 7Bcited by 3,164
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
38 more
  1. 2024Mixtral of Expertscited by 1,823
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  6. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  7. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  8. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  9. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  10. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  11. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  12. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  13. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  14. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  15. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 221
  16. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  17. 2018Constituency Parsing with a Self-Attentive Encodercited by 199
  18. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  19. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 181
  20. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 177
  21. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 174
  22. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  23. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  24. 2022mGPT: Few-Shot Learners Go Multilingualcited by 142
  25. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  26. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  27. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 89
  28. 2023Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptationcited by 81
  29. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  30. 2024Multilingual Instruction Tuning With Just a Pinch of Multilingualitycited by 73
  31. 2021Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translationcited by 70
  32. 2022NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languagescited by 68
  33. 2023LongForm: Effective Instruction Tuning with Reverse Instructionscited by 47
  34. 2023GlotLID: Language Identification for Low-Resource Languagescited by 39
  35. 2023ParroT: Translating during Chat using Large Language Models tuned with Human Translation and Feedbackcited by 34
  36. 2023Taxi1500: A Multilingual Dataset for Text Classification in 1500 Languagescited by 23
  37. 2023TarGEN: Targeted Data Generation with Large Language Modelscited by 17
  38. 2023DoG-Instruct: Towards Premium Instruction-Tuning Data via Text-Grounded Instruction Wrappingcited by 16

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 6.

12 papers.

  1. 2025Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuumcited by 18
  2. 2026F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual Worldcited by 12
  3. 2025Instruction Tuning on Public Government and Cultural Data for Low-Resource Language: a Case Study in Kazakhcited by 4
  4. 2025XL-Suite: Cross-Lingual Synthetic Training and Evaluation Data for Open-Ended Generationcited by 1
  5. 2025GemMaroc: Unlocking Darija Proficiency in LLMs with Minimal Datacited by 1
  6. 2025LuxInstruct: A Cross-Lingual Instruction Tuning Dataset For Luxembourgishcited by 1
  7. 2026Kakugo: Distillation of Low-Resource Languages into Small Language Modelscited by 1
  8. 2025LuxIT: A Luxembourgish Instruction Tuning Dataset from Monolingual Seed Datacited by 0
  9. 2026Toward Cross-Lingual Quality Classifiers for Multilingual Pretraining Data Selectioncited by 0
  10. 2026Why Low-Resource NLP Needs More Than Cross-Lingual Transfer: Lessons Learned from Luxembourgishcited by 0
  11. 2026ML-Embed: Inclusive and Efficient Embeddings for a Multilingual Worldcited by 0
  12. 2026EuroAlpaca: Task-Preserving Localisation of Instruction Data for European Languagescited by 0