# Tiny Aya: Bridging Scale and Multilingual Depth

arXiv: https://arxiv.org/abs/2603.11510 · HTML: https://mlsanity.com/paper/2603.11510/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 78 references that are in the map.
- First posted on arXiv 12 Mar 2026.

## Builds on (most cited first, 30 of 60)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models](https://mlsanity.com/paper/2308.01263/) (2023)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge](https://mlsanity.com/paper/2410.02736/) (2024)
- [The Impact of Positional Encoding on Length Generalization in Transformers](https://mlsanity.com/paper/2305.19466/) (2023)
- [Low-Resource Languages Jailbreak GPT-4](https://mlsanity.com/paper/2310.02446/) (2023)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2406.07791/) (2024)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [Multilingual Jailbreak Challenges in Large Language Models](https://mlsanity.com/paper/2310.06474/) (2023)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [Lessons from the Trenches on Reproducible Evaluation of Language Models](https://mlsanity.com/paper/2405.14782/) (2024)
- [BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages](https://mlsanity.com/paper/2406.09948/) (2024)
- [Towards Measuring and Modeling "Culture" in LLMs: A Survey](https://mlsanity.com/paper/2403.15412/) (2024)
- [Benchmarking Cognitive Biases in Large Language Models as Evaluators](https://mlsanity.com/paper/2309.17012/) (2023)
- [FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language](https://mlsanity.com/paper/2506.20920/) (2025)
- [Translation Artifacts in Cross-lingual Transfer Learning](https://mlsanity.com/paper/2004.04721/) (2020)

## Built on by (most cited first, 15 of 15)

- [Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling](https://mlsanity.com/paper/2604.25578/) (2026)
- [Defragmenting Language Models: An Interpretability-based Approach for Vocabulary Expansion](https://mlsanity.com/paper/2604.16656/) (2026)
- [MedAidDialog: A Multilingual Multi-Turn Medical Dialogue Dataset for Accessible Healthcare](https://mlsanity.com/paper/2603.24132/) (2026)
- [AFRILANGTUTOR: Advancing Language Tutoring and Culture Education in Low-Resource Languages with Large Language Models](https://mlsanity.com/paper/2604.20996/) (2026)
- [How Small Can You Go? LoRA Fine-Tuning 270M-8B Models for Merchant Information Extraction in Financial Transactions](https://mlsanity.com/paper/2606.08051/) (2026)
- [Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness](https://mlsanity.com/paper/2608.09766/) (2026)
- [Last Translation Benchmark](https://mlsanity.com/paper/2609.04173/) (2026)
- [AfrIFact: Cultural Information Retrieval, Evidence Extraction and Fact Checking for African Languages](https://mlsanity.com/paper/2604.00706/) (2026)
- [UFAL-CUNI at SemEval-2026 Task 11: An Efficient Modular Neuro-symbolic Method for Syllogistic Reasoning](https://mlsanity.com/paper/2605.04941/) (2026)
- [A Systematic Investigation of RL-Jailbreaking in LLMs](https://mlsanity.com/paper/2605.07032/) (2026)
- [AfriScience-MT: Towards Decolonizing Science in Africa through Text Translation](https://mlsanity.com/paper/2605.29741/) (2026)
- [Pancasila-Dilemmas: Evaluating Large Language Models on Indonesian Human Value Dilemmas Grounded in Pancasila](https://mlsanity.com/paper/2607.18066/) (2026)
- [Do LLMs Make More Mistakes If They Do Not Believe the Input Data?](https://mlsanity.com/paper/2609.09363/) (2026)
- [Building Multilingual Bridges: Data Mixing as the Pillar of Generalization for In-Language Reasoning](https://mlsanity.com/paper/2609.10445/) (2026)
- [Made in Hungary: Comments on the performance of generative language models](https://mlsanity.com/paper/2609.18284/) (2026)

## Abstract (arXiv)

Tiny Aya redefines what a small multilingual language model can achieve. Trained on 70 languages and refined through region-aware posttraining, it delivers state-of-the-art in translation quality, strong multilingual understanding, and high-quality target-language generation, all with just 3.35B parameters. The release includes a pretrained foundation model, a globally balanced instruction-tuned variant, and three region-specialized models targeting languages from Africa, South Asia, Europe, Asia-Pacific, and West Asia. This report details the training strategy, data composition, and comprehensive evaluation framework behind Tiny Aya, and presents an alternative scaling path for multilingual AI: one centered on efficiency, balanced performance across languages, and practical deployment.