mlsanityData to 14 Oct 2026

Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLM

arXiv 2512.21580 ↗First posted 25 Dec 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We present Gamayun, a 1.5B-parameter multilingual language model trained entirely from scratch on 2.5T tokens. Designed for efficiency and deployment in resource-constrained environments, Gamayun addresses the lack of research on small non-English-centric LLMs by adopting a novel two-stage pre-training strategy: balanced multilingual training for cross-lingual alignment, followed by high-quality English enrichment to transfer performance gains across languages. Our model supports 12 languages, with special focus on Russian. Despite a significantly smaller training budget than comparable models, Gamayun outperforms LLaMA3.2-1B (9T tokens) on all considered benchmarks, and surpasses Qwen2.5-1.5B (18T tokens) on a wide range of English and multilingual tasks. It matches or exceeds Qwen3 (36T tokens) on most tasks outside advanced STEM, achieving state-of-the-art results in Russian, including the MERA benchmark, among the models of comparable size (1-2B parameters).

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 62 of its 77 references that are in the map. First posted on arXiv 25 Dec 2025.

Builds on

62 of 77 references.

References · 62

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
38 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,512
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  6. 2025Gemma 3 Technical Reportcited by 1,824
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  8. 2020GLU Variants Improve Transformercited by 1,804
  9. 2019Root Mean Square Layer Normalizationcited by 1,480
  10. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  12. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  13. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  14. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  15. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  16. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  17. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  18. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  19. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  20. 2023The Falcon Series of Open Language Modelscited by 535
  21. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 511
  22. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  23. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  24. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  25. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  26. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
  27. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  28. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 232
  29. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 194
  30. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
  31. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  32. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  33. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 151
  34. 2022mGPT: Few-Shot Learners Go Multilingualcited by 142
  35. 2024EuroLLM: Multilingual Language Models for Europecited by 106
  36. 2024ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabiccited by 88
  37. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 86
  38. 2024SelfCodeAlign: Self-Alignment for Code Generationcited by 62

50 most cited of 62 All 62 (JSON)

Built on by

Not cited yet.