# Introducing cosmosGPT: Monolingual Training for Turkish Language Models

arXiv: https://arxiv.org/abs/2404.17336 · HTML: https://mlsanity.com/paper/2404.17336/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 26 Apr 2024.

## Builds on (most cited first, 14 of 14)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [OpenChat: Advancing Open-source Language Models with Mixed-Quality Data](https://mlsanity.com/paper/2309.11235/) (2023)
- [CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages](https://mlsanity.com/paper/2309.09400/) (2023)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation](https://mlsanity.com/paper/2305.15011/) (2023)
- [TURNA: A Turkish Encoder-Decoder Language Model for Enhanced Understanding and Generation](https://mlsanity.com/paper/2401.14373/) (2024)
- [SambaLingo: Teaching Large Language Models New Languages](https://mlsanity.com/paper/2404.05829/) (2024)
- [Mukayese: Turkish NLP Strikes Back](https://mlsanity.com/paper/2203.01215/) (2022)
- [Developing and Evaluating Tiny to Medium-Sized Turkish BERT Models](https://mlsanity.com/paper/2307.14134/) (2023)
- [VBART: The Turkish LLM](https://mlsanity.com/paper/2403.01308/) (2024)

## Built on by (most cited first, 8 of 8)

- [Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs](https://mlsanity.com/paper/2502.18795/) (2025)
- [TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs](https://mlsanity.com/paper/2506.13487/) (2025)
- [Tokens with Meaning: A Hybrid Tokenization Approach for Turkish](https://mlsanity.com/paper/2508.14292/) (2025)
- [LlamaTurk: Adapting Open-Source Generative Large Language Models for Low-Resource Language](https://mlsanity.com/paper/2405.07745/) (2024)
- [Optimizing Large Language Models for Turkish: New Methodologies in Corpus Selection and Training](https://mlsanity.com/paper/2412.02775/) (2024)
- [ufakzeka-1: Building and Evaluating a 151M-Parameter Turkish Language Model from Scratch](https://mlsanity.com/paper/2609.25081/) (2026)
- [Probing for Reading Times](https://mlsanity.com/paper/2604.18712/) (2026)
- [Complexity-Guided Component-wise Initialization for Language Model Pretraining](https://mlsanity.com/paper/2607.09204/) (2026)

## Abstract (arXiv)

The number of open source language models that can produce Turkish is increasing day by day, as in other languages. In order to create the basic versions of such models, the training of multilingual models is usually continued with Turkish corpora. The alternative is to train the model with only Turkish corpora. In this study, we first introduce the cosmosGPT models that we created with this alternative method. Then, we introduce new finetune datasets for basic language models to fulfill user requests and new evaluation datasets for measuring the capabilities of Turkish language models. Finally, a comprehensive comparison of the adapted Turkish language models on different capabilities is presented. The results show that the language models we built with the monolingual corpus have promising performance despite being about 10 times smaller than the others.