mlsanityData to 7 Oct 2026

Tiny Aya: Bridging Scale and Multilingual Depth

arXiv 2603.11510 ↗First posted 12 Mar 2026cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Tiny Aya redefines what a small multilingual language model can achieve. Trained on 70 languages and refined through region-aware posttraining, it delivers state-of-the-art in translation quality, strong multilingual understanding, and high-quality target-language generation, all with just 3.35B parameters. The release includes a pretrained foundation model, a globally balanced instruction-tuned variant, and three region-specialized models targeting languages from Africa, South Asia, Europe, Asia-Pacific, and West Asia. This report details the training strategy, data composition, and comprehensive evaluation framework behind Tiny Aya, and presents an alternative scaling path for multilingual AI: one centered on efficiency, balanced performance across languages, and practical deployment.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 60 of its 78 references that are in the map. First posted on arXiv 12 Mar 2026.

Builds on

60 of 78 references.

References · 60

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  6. 2020GLU Variants Improve Transformercited by 1,724
  7. 2025Gemma 3 Technical Reportcited by 1,686
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  10. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  11. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  12. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
38 more
  1. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  2. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  3. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  4. 2023Scaling Data-Constrained Language Modelscited by 391
  5. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  6. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  7. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  8. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  9. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  10. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  11. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  12. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  13. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  14. 2024BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languagescited by 171
  15. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 166
  16. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  17. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 134
  18. 2020Translation Artifacts in Cross-lingual Transfer Learningcited by 95
  19. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 92
  20. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
  21. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 67
  22. 2022Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP Worldcited by 55
  23. 2025PolyMath: Evaluating Mathematical Reasoning in Multilingual Contextscited by 52
  24. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settingscited by 50
  25. 2024NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Modelscited by 47
  26. 2024To Code, or Not To Code? Exploring Impact of Code in Pre-trainingcited by 47
  27. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  28. 2024IrokoBench: A New Benchmark for African Languages in the Age of Large Language Modelscited by 43
  29. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 42
  30. 2024How Does Quantization Affect Multilingual LLMs?cited by 41
  31. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  32. 2023AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languagescited by 24
  33. 2024Scaling Laws for Multilingual Language Modelscited by 22
  34. 2025SEA-LION: Southeast Asian Languages in One Networkcited by 17
  35. 2024EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Modelscited by 16
  36. 2024The Zeno's Paradox of `Low-Resource' Languagescited by 12
  37. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 11
  38. 2025Culture is Everywhere: A Call for Intentionally Cultural Evaluationcited by 10

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 14.

14 papers.

Sort:

By area:

  1. 2026Defragmenting Language Models: An Interpretability-based Approach for Vocabulary Expansioncited by 2
  2. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcyclingcited by 2
  3. 2026MedAidDialog: A Multilingual Multi-Turn Medical Dialogue Dataset for Accessible Healthcarecited by 1
  4. 2026AFRILANGTUTOR: Advancing Language Tutoring and Culture Education in Low-Resource Languages with Large Language Modelscited by 1
  5. 2026How Small Can You Go? LoRA Fine-Tuning 270M-8B Models for Merchant Information Extraction in Financial Transactionscited by 1
  6. 2026Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustnesscited by 1
  7. 2026Last Translation Benchmarkcited by 1
  8. 2026AfrIFact: Cultural Information Retrieval, Evidence Extraction and Fact Checking for African Languagescited by 0
  9. 2026UFAL-CUNI at SemEval-2026 Task 11: An Efficient Modular Neuro-symbolic Method for Syllogistic Reasoningcited by 0
  10. 2026A Systematic Investigation of RL-Jailbreaking in LLMscited by 0
  11. 2026AfriScience-MT: Towards Decolonizing Science in Africa through Text Translationcited by 0
  12. 2026Do LLMs Make More Mistakes If They Do Not Believe the Input Data?cited by 0
2 more
  1. 2026Building Multilingual Bridges: Data Mixing as the Pillar of Generalization for In-Language Reasoningcited by 0
  2. 2026Made in Hungary: Comments on the performance of generative language modelscited by 0