mlsanityData to 7 Oct 2026

Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning

arXiv 2402.06619 ↗First posted 9 Feb 2024cs.CL cs.AIcited by 174

Not on the map. Why?

Abstract (arXiv)

Datasets are foundational to many breakthroughs in modern artificial intelligence. Many recent achievements in the space of natural language processing (NLP) can be attributed to the finetuning of pre-trained models on a diverse set of tasks that enables a large language model (LLM) to respond to instructions. Instruction fine-tuning (IFT) requires specifically constructed and annotated datasets. However, existing datasets are almost all in the English language. In this work, our primary goal is to bridge the language gap by building a human-curated instruction-following dataset spanning 65 languages. We worked with fluent speakers of languages from around the world to collect natural instances of instructions and completions. Furthermore, we create the most extensive multilingual collection to date, comprising 513 million instances through templating and translating existing datasets across 114 languages. In total, we contribute four key resources: we develop and open-source the Aya Annotation Platform, the Aya Dataset, the Aya Collection, and the Aya Evaluation Suite. The Aya initiative also serves as a valuable case study in participatory research, involving collaborators from 119 countries. We see this as a valuable framework for future research collaborations that aim to bridge gaps in resources.

Key facts. Cited by 174 papers in the map (data through 7 Oct 2026). Builds on 116 of its 215 references that are in the map. First posted on arXiv 9 Feb 2024.

Builds on

116 of 215 references.

References · 116

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  12. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
38 more
  1. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  2. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  3. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  4. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  5. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  6. 2023LIMA: Less Is More for Alignmentcited by 1,080
  7. 2015Teaching Machines to Read and Comprehendcited by 1,039
  8. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  9. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  10. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  11. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  12. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  13. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  14. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  15. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  16. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  17. 2018CoQA: A Conversational Question Answering Challengecited by 674
  18. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  19. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  20. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  21. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  22. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  23. 2021MetaICL: Learning to Learn In Contextcited by 450
  24. 2018Deep Learning for Classical Japanese Literaturecited by 415
  25. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  26. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  27. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  28. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  29. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  30. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  31. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  32. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  33. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  34. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  35. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  36. 2023Poisoning Language Models During Instruction Tuningcited by 254
  37. 2019Massively Multilingual Neural Machine Translationcited by 246
  38. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241

50 most cited of 116 All 116 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 85; 2025: 49; 2026: 40.

174 papers.

Sort:

By area:

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  2. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  3. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  4. 2024Generative Representational Instruction Tuningcited by 227
  5. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
  6. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 121
  7. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 121
  8. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  9. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  10. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 78
  11. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
  12. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
38 more
  1. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 67
  2. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 65
  3. 2024To Code, or Not To Code? Exploring Impact of Code in Pre-trainingcited by 47
  4. 2024IrokoBench: A New Benchmark for African Languages in the Age of Large Language Modelscited by 43
  5. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 42
  6. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 41
  7. 2024How Does Quantization Affect Multilingual LLMs?cited by 41
  8. 2024Instruction Tuning With Loss Over Instructionscited by 40
  9. 2024PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Datacited by 37
  10. 2024Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personascited by 36
  11. 2024X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scalecited by 36
  12. 2025KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Modelcited by 35
  13. 2025Large-Scale Data Selection for Instruction Tuningcited by 34
  14. 2024A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?cited by 33
  15. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 33
  16. 2025The Bitter Lesson Learned from 2,000+ Multilingual Benchmarkscited by 31
  17. 2025MergeBench: A Benchmark for Merging Domain-Specialized LLMscited by 30
  18. 2025SafeSwitch: Steering Unsafe LLM Behavior via Internal Activation Signalscited by 29
  19. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
  20. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignmentcited by 26
  21. 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 26
  22. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  23. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  24. 2025Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoningcited by 23
  25. 2025BOUQuET: dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translationcited by 21
  26. 2024High-Dimension Human Value Representation in Large Language Modelscited by 20
  27. 2024Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelscited by 20
  28. 2024Sailor: Open Language Models for South-East Asiacited by 19
  29. 2024Unlocking the Potential of Model Merging for Low-Resource Languagescited by 19
  30. 2024Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMscited by 19
  31. 2025Localized Cultural Knowledge is Conserved and Controllable in Large Language Modelscited by 19
  32. 2024From One to Many: Expanding the Scope of Toxicity Mitigation in Language Modelscited by 18
  33. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  34. 2024On the Limitations of Compute Thresholds as a Governance Strategycited by 17
  35. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 17
  36. 2024EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Modelscited by 16
  37. 2024UTF:Undertrained Tokens as Fingerprints A Novel Approach to LLM Identificationcited by 16
  38. 2025Palm: A Culturally Inclusive and Linguistically Diverse Dataset for Arabic LLMscited by 15

50 most cited of 174 All 174 (JSON)