# TurkBench: A Benchmark for Evaluating Turkish Large Language Models

arXiv: https://arxiv.org/abs/2601.07020 · HTML: https://mlsanity.com/paper/2601.07020/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 39 references that are in the map.
- First posted on arXiv 11 Jan 2026.

## Builds on (most cited first, 21 of 21)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation](https://mlsanity.com/paper/2004.01401/) (2020)
- [LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback](https://mlsanity.com/paper/2406.01771/) (2024)
- [CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractions](https://mlsanity.com/paper/2012.04293/) (2020)
- [TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish](https://mlsanity.com/paper/2407.12402/) (2024)
- [Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmark](https://mlsanity.com/paper/2405.20574/) (2024)
- [Evaluating the Quality of Benchmark Datasets for Low-Resource Languages: A Case Study on Turkish](https://mlsanity.com/paper/2504.09714/) (2025)
- [Bridging the Bosphorus: Advancing Turkish Large Language Models through Strategies for Low-Resource Language Adaptation and Benchmarking](https://mlsanity.com/paper/2405.04685/) (2024)
- [Cetvel: A Unified Benchmark for Evaluating Language Understanding, Generation and Cultural Capacity of LLMs for Turkish](https://mlsanity.com/paper/2508.16431/) (2025)
- [Resources for Turkish Natural Language Processing: A critical survey](https://mlsanity.com/paper/2204.05042/) (2022)
- [Benchmarking Procedural Language Understanding for Low-Resource Languages: A Case Study on Turkish](https://mlsanity.com/paper/2309.06698/) (2023)
- [B"uy"uk Dil Modelleri iccin TR-MMLU Benchmarki: Performans Deugerlendirmesi, Zorluklar ve .Iyilecstirme Firsatlari](https://mlsanity.com/paper/2508.13044/) (2025)

## Built on by (most cited first, 5 of 5)

- [TurkicNLP: An NLP Toolkit for Turkic Languages](https://mlsanity.com/paper/2602.19174/) (2026)
- [GaelEval: Benchmarking LLM Performance for Scottish Gaelic](https://mlsanity.com/paper/2604.02135/) (2026)
- [HakemBench: A Turkish Benchmark of Typed Decisions](https://mlsanity.com/paper/2610.02293/) (2026)
- [Turkish MMLU Pro: Traceable Option Augmentation and Its Validity Limits in Turkish Multiple-Choice Evaluation](https://mlsanity.com/paper/2609.15467/) (2026)
- [Does CoT-Pass@k Really Check the CoT? A Multilingual Mathematical Audit](https://mlsanity.com/paper/2609.32622/) (2026)

## Abstract (arXiv)

With the recent surge in the development of large language models, the need for comprehensive and language-specific evaluation benchmarks has become critical. While significant progress has been made in evaluating English-language models, benchmarks for other languages, particularly those with unique linguistic characteristics such as Turkish, remain less developed. Our study introduces TurkBench, a comprehensive benchmark designed to assess the capabilities of generative large language models in the Turkish language. TurkBench involves 8,151 data samples across 21 distinct subtasks. These are organized under six main categories of evaluation: Knowledge, Language Understanding, Reasoning, Content Moderation, Turkish Grammar and Vocabulary, and Instruction Following. The diverse range of tasks and the culturally relevant data would provide researchers and developers with a valuable tool for evaluating their models and identifying areas for improvement. We further publish our benchmark for online submissions at https://huggingface.co/turkbench