mlsanityData to 7 Oct 2026

Aya 23: Open Weight Releases to Further Multilingual Progress

arXiv 2405.15032 ↗First posted 23 May 2024cs.CLcited by 121

Not on the map. Why?

Abstract (arXiv)

This technical report introduces Aya 23, a family of multilingual language models. Aya 23 builds on the recent release of the Aya model ("Ust"un et al., 2024), focusing on pairing a highly performant pre-trained model with the recently released Aya collection (Singh et al., 2024). The result is a powerful multilingual large language model serving 23 languages, expanding state-of-art language modeling capabilities to approximately half of the world's population. The Aya model covered 101 languages whereas Aya 23 is an experiment in depth vs breadth, exploring the impact of allocating more capacity to fewer languages that are included during pre-training. Aya 23 outperforms both previous massively multilingual models like Aya 101 for the languages it covers, as well as widely used models like Gemma, Mistral and Mixtral on an extensive range of discriminative and generative tasks. We release the open weights for both the 8B and 35B models as part of our continued commitment for expanding access to multilingual progress.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). Builds on 45 of its 59 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

45 of 59 references.

References · 45

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  10. 2020GLU Variants Improve Transformercited by 1,724
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
33 more
  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  2. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  3. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  4. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  5. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  6. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  7. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  8. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  9. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  10. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  11. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  12. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  13. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  14. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  15. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  16. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  17. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  18. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  19. 2023Gender bias and stereotypes in Large Language Modelscited by 185
  20. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  21. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  22. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 94
  23. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  24. 2023Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptationcited by 79
  25. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  26. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  27. 2023AfroBench: How Good are Large Language Models on African Languages?cited by 43
  28. 2023GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLPcited by 37
  29. 2023Lost in Translation: Large Language Models in Non-English Content Analysiscited by 34
  30. 2023On Evaluating and Mitigating Gender Biases in Multilingual Settingscited by 31
  31. 2023A Material Lens on Coloniality in NLPcited by 19
  32. 2023Indian-BhED: A Dataset for Measuring India-Centric Biases in Large Language Modelscited by 17
  33. 2022Scalable Training of Language Models using JAX pjit and TPUv4cited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 53; 2025: 51; 2026: 17.

121 papers.

Sort:

By area:

  1. 2024BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languagescited by 171
  2. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  3. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  4. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
  5. 2025MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluationcited by 73
  6. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settingscited by 50
  7. 2025MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learningcited by 48
  8. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 47
  9. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 42
  10. 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformerscited by 42
  11. 2024How Does Quantization Affect Multilingual LLMs?cited by 41
  12. 2025Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gapcited by 41
38 more
  1. 2024X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scalecited by 36
  2. 2024SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languagescited by 35
  3. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
  4. 2025Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoningcited by 33
  5. 2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Docited by 32
  6. 2024TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkishcited by 31
  7. 2024LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languagescited by 28
  8. 2024CUTE: Measuring LLMs' Understanding of Their Tokenscited by 28
  9. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 27
  10. 2024INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languagescited by 24
  11. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  12. 2025Multiple LLM Agents Debate for Equitable Cultural Alignmentcited by 20
  13. 2025Enhancing LLM Safety Through a Theoretical Minimax Game Lenscited by 18
  14. 2025Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenizationcited by 18
  15. 2024WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language Modelscited by 17
  16. 2024Cross-Lingual Auto Evaluation for Assessing Multilingual LLMscited by 17
  17. 2024EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Modelscited by 16
  18. 2024Evaluating Morphological Compositional Generalization in Large Language Modelscited by 16
  19. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  20. 2024MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systemscited by 14
  21. 2024On the Implications of Verbose LLM Outputs: A Case Study in Translation Evaluationcited by 13
  22. 2025SSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translationcited by 13
  23. 2024Tucano: Advancing Neural Text Generation for Portuguesecited by 12
  24. 2024HateDay: Insights from a Global Hate Speech Dataset Representative of a Day on Twittercited by 12
  25. 2024MINERS: Multilingual Language Models as Semantic Retrieverscited by 11
  26. 2024Ladder: A Model-Agnostic Framework Boosting LLM-based Machine Translation to the Next Levelcited by 11
  27. 2025Improving Multilingual Language Models by Aligning Representations through Steeringcited by 11
  28. 2024CamelEval: Advancing Culturally Aligned Arabic Language Models and Benchmarkscited by 10
  29. 2025AfriHate: A Multilingual Collection of Hate Speech and Abusive Language Datasets for African Languagescited by 10
  30. 2025Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindicited by 10
  31. 2024Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Modelscited by 9
  32. 2024FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Datacited by 9
  33. 2025HKCanto-Eval: A Benchmark for Evaluating Cantonese Language Understanding and Cultural Comprehension in LLMscited by 8
  34. 2024Uhura: A Benchmark for Evaluating Scientific Question Answering and Truthfulness in Low-Resource African Languagescited by 7
  35. 2025Continually Adding New Languages to Multilingual Language Modelscited by 7
  36. 2024LLM See, LLM Do: Guiding Data Generation to Target Non-Differentiable Objectivescited by 6
  37. 2024On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruningcited by 6
  38. 2024Multilingual Arbitrage: Optimizing Data Pools to Accelerate Multilingual Progresscited by 6

50 most cited of 121 All 121 (JSON)