# Tokenization Standards for Linguistic Integrity: Turkish as a Benchmark

arXiv: https://arxiv.org/abs/2502.07057 · HTML: https://mlsanity.com/paper/2502.07057/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 20 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Builds on (most cited first, 7 of 7)

- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [EuroLLM: Multilingual Language Models for Europe](https://mlsanity.com/paper/2409.16235/) (2024)
- [Tokenization Is More Than Compression](https://mlsanity.com/paper/2402.18376/) (2024)
- [How Much Does Tokenization Affect Neural Machine Translation?](https://mlsanity.com/paper/1812.08621/) (2018)
- [How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese](https://mlsanity.com/paper/2306.09572/) (2023)
- [Setting Standards in Turkish NLP: TR-MMLU for Large Language Model Evaluation](https://mlsanity.com/paper/2501.00593/) (2024)

## Built on by (most cited first, 8 of 8)

- [Tokenization Matters: Improving Zero-Shot NER for Indic Languages](https://mlsanity.com/paper/2504.16977/) (2025)
- [Tokens with Meaning: A Hybrid Tokenization Approach for Turkish](https://mlsanity.com/paper/2508.14292/) (2025)
- [UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?](https://mlsanity.com/paper/2508.11260/) (2025)
- [Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domain](https://mlsanity.com/paper/2601.16018/) (2026)
- [MoganBert-TR: A Turkish Encoder Foundation Model Trained from Scratch with a CLM-to-MLM Curriculum](https://mlsanity.com/paper/2608.25768/) (2026)
- [Plausibility as Commonsense Reasoning: Humans Succeed, Large Language Models Do not](https://mlsanity.com/paper/2604.04825/) (2026)
- [Benchmarking Source-Sensitive Reasoning in Turkish: Humans and LLMs under Evidential Trust Manipulation](https://mlsanity.com/paper/2604.24665/) (2026)
- [Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish](https://mlsanity.com/paper/2606.18717/) (2026)

## Abstract (arXiv)

Tokenization is a fundamental preprocessing step in NLP, directly impacting large language models' (LLMs) ability to capture syntactic, morphosyntactic, and semantic structures. This paper introduces a novel framework for systematically evaluating tokenization strategies, addressing challenges in morphologically rich and low-resource languages. Using a Turkish dataset of 6,200 multiple-choice questions from the Massive Multitask Language Understanding (MMLU) benchmark, the framework assesses tokenizers across five key metrics: vocabulary size, token count, processing time, language-specific token percentages (%TR), and token purity. These metrics provide a structured approach to evaluating how well tokenizers preserve linguistic structures. While %TR measures the proportion of valid words in the target language, %Pure assesses the alignment of tokens with meaningful linguistic units, such as roots and valid morphemes, minimizing semantic fragmentation. The findings reveal that %TR, introduced as a critical metric, exhibits a stronger correlation with downstream performance (e.g., MMLU scores) than token purity, emphasizing its role in improving model accuracy. Additionally, larger model parameters do not necessarily yield better tokenization quality or enhanced results, highlighting the importance of tailored tokenization strategies that prioritize linguistic alignment. This framework sets a new standard for developing robust tokenization methods optimized for morphologically complex and low-resource languages. Future work will refine morphological analysis, explore domain-specific customizations, and conduct cross-linguistic evaluations to further enhance tokenization practices.