# Bridging the Bosphorus: Advancing Turkish Large Language Models through Strategies for Low-Resource Language Adaptation and Benchmarking

arXiv: https://arxiv.org/abs/2405.04685 · HTML: https://mlsanity.com/paper/2405.04685/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 48 references that are in the map.
- First posted on arXiv 7 May 2024.

## Builds on (most cited first, 30 of 32)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](https://mlsanity.com/paper/2201.06642/) (2022)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages](https://mlsanity.com/paper/2006.06202/) (2020)
- [CroissantLLM: A Truly Bilingual French-English Language Model](https://mlsanity.com/paper/2402.00786/) (2024)

## Built on by (most cited first, 10 of 10)

- [TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish](https://mlsanity.com/paper/2407.12402/) (2024)
- [Script Fragmentation and Format: What Drives the English-Bengali Performance Gap in Open LLMs?](https://mlsanity.com/paper/2507.23248/) (2025)
- [Overcoming Data Scarcity in Generative Language Modelling for Low-Resource Languages: A Systematic Review](https://mlsanity.com/paper/2505.04531/) (2025)
- [Cetvel: A Unified Benchmark for Evaluating Language Understanding, Generation and Cultural Capacity of LLMs for Turkish](https://mlsanity.com/paper/2508.16431/) (2025)
- [LlamaTurk: Adapting Open-Source Generative Large Language Models for Low-Resource Language](https://mlsanity.com/paper/2405.07745/) (2024)
- [TurkBench: A Benchmark for Evaluating Turkish Large Language Models](https://mlsanity.com/paper/2601.07020/) (2026)
- [Make Satire Boring Again: Reducing Stylistic Bias of Satirical Corpus by Utilizing Generative LLMs](https://mlsanity.com/paper/2412.09247/) (2024)
- [Beyond Task-Oriented and Chitchat Dialogues: Proactive and Transition-Aware Conversational Agents](https://mlsanity.com/paper/2511.08835/) (2025)
- [Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domain](https://mlsanity.com/paper/2601.16018/) (2026)
- [Achieving Trustworthy Real-Time Decision Support Systems with Low-Latency Interpretable AI Models](https://mlsanity.com/paper/2506.20018/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) are becoming crucial across various fields, emphasizing the urgency for high-quality models in underrepresented languages. This study explores the unique challenges faced by low-resource languages, such as data scarcity, model selection, evaluation, and computational limitations, with a special focus on Turkish. We conduct an in-depth analysis to evaluate the impact of training strategies, model choices, and data availability on the performance of LLMs designed for underrepresented languages. Our approach includes two methodologies: (i) adapting existing LLMs originally pretrained in English to understand Turkish, and (ii) developing a model from the ground up using Turkish pretraining data, both supplemented with supervised fine-tuning on a novel Turkish instruction-tuning dataset aimed at enhancing reasoning capabilities. The relative performance of these methods is evaluated through the creation of a new leaderboard for Turkish LLMs, featuring benchmarks that assess different reasoning and knowledge skills. Furthermore, we conducted experiments on data and model scaling, both during pretraining and fine-tuning, simultaneously emphasizing the capacity for knowledge transfer across languages and addressing the challenges of catastrophic forgetting encountered during fine-tuning on a different language. Our goal is to offer a detailed guide for advancing the LLM framework in low-resource linguistic contexts, thereby making natural language processing (NLP) benefits more globally accessible.