mlsanityData to 7 Oct 2026

Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model

arXiv 2402.07827 ↗First posted 12 Feb 2024cs.CLcited by 292

Not on the map. Why?

Abstract (arXiv)

Recent breakthroughs in large language models (LLMs) have centered around a handful of data-rich languages. What does it take to broaden access to breakthroughs beyond first-class citizen languages? Our work introduces Aya, a massively multilingual generative language model that follows instructions in 101 languages of which over 50% are considered as lower-resourced. Aya outperforms mT0 and BLOOMZ on the majority of tasks while covering double the number of languages. We introduce extensive new evaluation suites that broaden the state-of-art for multilingual eval across 99 languages -- including discriminative and generative tasks, human evaluation, and simulated win rates that cover both held-out tasks and in-distribution performance. Furthermore, we conduct detailed investigations on the optimal finetuning mixture composition, data pruning, as well as the toxicity, bias, and safety of our models. We open-source our instruction datasets and our model at https://hf.co/CohereForAI/aya-101

Key facts. Cited by 292 papers in the map (data through 7 Oct 2026). Builds on 119 of its 158 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

119 of 158 references.

References · 119

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
38 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  9. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  11. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  12. 2023LIMA: Less Is More for Alignmentcited by 1,080
  13. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  14. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  15. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  16. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  17. 2023StarCoder: may the source be with you!cited by 790
  18. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  19. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  20. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  21. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  22. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  23. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  24. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  25. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  26. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  27. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  28. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  29. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  30. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  31. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  32. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  33. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  34. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  35. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  36. 2019The State of Sparsity in Deep Neural Networkscited by 471
  37. 2021MetaICL: Learning to Learn In Contextcited by 450
  38. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419

50 most cited of 119 All 119 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 130; 2025: 110; 2026: 52.

292 papers.

Sort:

By area:

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  2. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  3. 2024Generative Representational Instruction Tuningcited by 227
  4. 2024BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languagescited by 171
  5. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 121
  6. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  7. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  8. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 101
  9. 2024Scaling Laws for Precisioncited by 92
  10. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  11. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 78
  12. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
38 more
  1. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 67
  2. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 65
  3. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedbackcited by 51
  4. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 47
  5. 2024To Code, or Not To Code? Exploring Impact of Code in Pre-trainingcited by 47
  6. 2024AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMscited by 46
  7. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  8. 2024Consent in Crisis: The Rapid Decline of the AI Data Commonscited by 45
  9. 2024IrokoBench: A New Benchmark for African Languages in the Age of Large Language Modelscited by 43
  10. 2024Latxa: An Open Language Model and Evaluation Suite for Basquecited by 42
  11. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 42
  12. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 41
  13. 2024How Does Quantization Affect Multilingual LLMs?cited by 41
  14. 2025Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMscited by 39
  15. 2024Digital Forgetting in Large Language Models: A Survey of Unlearning Methodscited by 38
  16. 2025Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Studycited by 38
  17. 2024PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Datacited by 37
  18. 2024Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanesecited by 36
  19. 2024X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scalecited by 36
  20. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  21. 2024MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMscited by 32
  22. 2024mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generationcited by 32
  23. 2024Preference Tuning For Toxicity Mitigation Generalizes Across Languagescited by 31
  24. 2024TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkishcited by 31
  25. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
  26. 2024Language Imbalance Driven Rewarding for Multilingual Self-improvingcited by 27
  27. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 27
  28. 2024Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Modelcited by 25
  29. 2024Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Interventioncited by 24
  30. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  31. 2024Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?cited by 22
  32. 2024Exploring Design Choices for Building Language-Specific LLMscited by 22
  33. 2025Presumed Cultural Identity: How Names Shape LLM Responsescited by 21
  34. 2024High-Dimension Human Value Representation in Large Language Modelscited by 20
  35. 2024Egalitarian Language Representation in Language Models: It All Begins with Tokenizerscited by 20
  36. 2024Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMscited by 20
  37. 2025The Emergence of Abstract Thought in Large Language Models Beyond Any Languagecited by 20
  38. 2025Localized Cultural Knowledge is Conserved and Controllable in Large Language Modelscited by 19

50 most cited of 292 All 292 (JSON)