mlsanityData to 7 Oct 2026

From Zero to Hero: An Open LLM Ecosystem for Armenian

arXiv 2609.03350 ↗First posted 3 Sep 2026cs.LG cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Pretraining data for Armenian, a morphologically rich and low-resource language, is scarce, and no open Armenian LLM has been released with the data and recipe needed to reproduce it. To address this gap, we curate and release two datasets. ArmWeb is an extensively validated corpus of 4.37M Armenian news documents. ArmSTEM is a parallel English-Armenian collection of 373K math and science problems with step-by-step solutions, translated into Armenian and verified through both answer-preserving LLM judgment and human evaluation. Continued pretraining of Gemma-4-E4B on these datasets yields arm-gemma-e4b, which outperforms every existing open Armenian model as well as its unadapted base, and is the first open Armenian LLM with complete training data and recipe. Our ablations show that news-only continued pretraining improves fluency while eroding knowledge, a pattern we also observe in existing Armenian models, and that a small share of verified translated STEM data reverses the loss. We further find that the largest public Armenian corpora overlap web-derived evaluation panels heavily, including a train/test self-overlap inside FineWeb-2. We openly release all data, models, and code.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 58 of its 88 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

58 of 88 references.

References · 58

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  11. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  12. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
38 more
  1. 2018Datasheets for Datasetscited by 1,371
  2. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  3. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  4. 2018Experience Replay for Continual Learningcited by 1,074
  5. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  6. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  7. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  8. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  9. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  10. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  11. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  12. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  13. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  14. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  15. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  16. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  17. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
  18. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 194
  19. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  20. 2018Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translationcited by 181
  21. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 181
  22. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  23. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 147
  24. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  25. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  26. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 134
  27. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  28. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  29. 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
  30. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 94
  31. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  32. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 78
  33. 2019Training Neural Machine Translation To Apply Terminology Constraintscited by 73
  34. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 69
  35. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 65
  36. 2024Latxa: An Open Language Model and Evaluation Suite for Basquecited by 42
  37. 2025DataDecide: How to Predict Best Pretraining Data with Small Experimentscited by 36
  38. 2024Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLPcited by 35

50 most cited of 58 All 58 (JSON)

Built on by

Not cited yet.