mlsanityData to 14 Oct 2026

ufakzeka-1: Building and Evaluating a 151M-Parameter Turkish Language Model from Scratch

arXiv 2609.25081 ↗First posted 18 Sep 2026cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

We describe ufakzeka-1, a 151M-parameter (182M with embeddings) decoder-only Turkish language model pretrained from scratch on 13.5B tokens of openly licensed text and instruction-tuned for chat, at a total cost of about 286 in cloud GPU, API and notebook time. The contribution is not the model's capability, which is what a model this size can be expected to have, but the record of building and measuring it: a Turkish byte-level tokenizer at 1.77 tokens per word, a three-stage pretraining schedule, a post-training mixture of openly licensed and generated data, and an evaluation battery of release gates, a rule-checked sweep of 5,508 conversations, judged conversations and hand tests, all with prompts held out from the training data, enforced by decontamination inside the data build and by a checked-in invariant script we run before each build. We report three findings that we believe transfer to other small-model efforts: a safety gate that had been "fixed" with training data written from its own questions read 64/64 while the honest figure was 34/64; training-seed variance was as large as the spread across every recipe we tried, so single-seed comparisons at this scale are uninformative; and data rounds repaired only what was absent from the data, while identity tracking over long context and multi-turn arithmetic did not move across any data change we tried, which we read as limits of the model size rather than gaps in the data, a reading the next, larger model will test. Weights, the data recipe, the evaluation code and the spend ledger are released under Apache-2.0.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 35 of its 39 references that are in the map. First posted on arXiv 18 Sep 2026.

Builds on

35 of 39 references.

References · 35

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  12. 2020GLU Variants Improve Transformercited by 1,804
23 more
  1. 2019Root Mean Square Layer Normalizationcited by 1,480
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  3. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  4. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  5. 2025Muon is Scalable for LLM Trainingcited by 428
  6. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  7. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  8. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  9. 2020Query-Key Normalization for Transformerscited by 280
  10. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  11. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  12. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 177
  13. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 143
  14. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 115
  15. 2024Number Cookbook: Number Understanding of Language Models and How to Improve Itcited by 42
  16. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 35
  17. 2024TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkishcited by 32
  18. 2026Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimizationcited by 16
  19. 2026Replaying pre-training data improves fine-tuningcited by 13
  20. 2025TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairscited by 9
  21. 2024Introducing cosmosGPT: Monolingual Training for Turkish Language Modelscited by 8
  22. 2026When Small Models Are Right for Wrong Reasons: Process Verification for Trustworthy Agentscited by 3
  23. 2026The Readout Shortcut: Positional Number Copying Dominates Arithmetic CoT Readout in Small Language Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.