# TurkicNLP: An NLP Toolkit for Turkic Languages

arXiv: https://arxiv.org/abs/2602.19174 · HTML: https://mlsanity.com/paper/2602.19174/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 102 references that are in the map.
- First posted on arXiv 22 Feb 2026.

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages](https://mlsanity.com/paper/2305.12182/) (2023)
- [Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing](https://mlsanity.com/paper/2101.03289/) (2021)
- [Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages](https://mlsanity.com/paper/2205.12654/) (2022)
- [TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish](https://mlsanity.com/paper/2407.12402/) (2024)
- [PyThaiNLP: Thai Natural Language Processing in Python](https://mlsanity.com/paper/2312.04649/) (2023)
- [A Large-Scale Study of Machine Translation in the Turkic Languages](https://mlsanity.com/paper/2109.04593/) (2021)
- [Polyglot: Distributed Word Representations for Multilingual NLP](https://mlsanity.com/paper/1307.1662/) (2013)
- [KazNERD: Kazakh Named Entity Recognition Dataset](https://mlsanity.com/paper/2111.13419/) (2021)
- [TURNA: A Turkish Encoder-Decoder Language Model for Enhanced Understanding and Generation](https://mlsanity.com/paper/2401.14373/) (2024)
- [Cross-Lingual Word Embeddings for Turkic Languages](https://mlsanity.com/paper/2005.08340/) (2020)
- [Setting Standards in Turkish NLP: TR-MMLU for Large Language Model Evaluation](https://mlsanity.com/paper/2501.00593/) (2024)
- [KazakhTTS2: Extending the Open-Source Kazakh TTS Corpus With More Data, Speakers, and Topics](https://mlsanity.com/paper/2201.05771/) (2022)
- [Mukayese: Turkish NLP Strikes Back](https://mlsanity.com/paper/2203.01215/) (2022)
- [TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages](https://mlsanity.com/paper/2502.11020/) (2025)
- [Evaluating Multiway Multilingual NMT in the Turkic Languages](https://mlsanity.com/paper/2109.06262/) (2021)
- [Cetvel: A Unified Benchmark for Evaluating Language Understanding, Generation and Cultural Capacity of LLMs for Turkish](https://mlsanity.com/paper/2508.16431/) (2025)
- [KazQAD: Kazakh Open-Domain Question Answering Dataset](https://mlsanity.com/paper/2404.04487/) (2024)
- [Recent Advancements and Challenges of Turkic Central Asian Language Processing](https://mlsanity.com/paper/2407.05006/) (2024)
- [KazParC: Kazakh Parallel Corpus for Machine Translation](https://mlsanity.com/paper/2403.19399/) (2024)
- [TurkBench: A Benchmark for Evaluating Turkish Large Language Models](https://mlsanity.com/paper/2601.07020/) (2026)
- [KyrgyzNLP: Challenges, Progress, and Future](https://mlsanity.com/paper/2411.05503/) (2024)
- [Automatically Annotated Turkish Corpus for Named Entity Recognition and Text Categorization using Large-Scale Gazetteers](https://mlsanity.com/paper/1702.02363/) (2017)

## Built on by (most cited first, 1 of 1)

- [Universal or Language-Family-Specific Script Unification for Cross-Lingual Transfer? A Case Study on Turkic Languages](https://mlsanity.com/paper/2608.09356/) (2026)

## Abstract (arXiv)

Turkic languages are spoken by about 200 million people and are written in Latin, Cyrillic, and Perso-Arabic scripts, yet the NLP tools that exist for them are scattered across languages, scripts, and incompatible software. We present TurkicNLP, an open-source Python library that offers a single pipeline for 24 Turkic languages, covering tokenization, morphological analysis, POS tagging, dependency parsing, named entity recognition, language identification, sentence embeddings, and machine translation. The library is designed for cross-lingual use: it detects and converts between scripts, maps 23 languages into the shared Common Turkic Alphabet, and includes multilingual models that share parameters across the family and transfer to languages without annotated data. Our evaluation shows the benefits of this design. The multilingual parser outperforms per-language parsers, the multilingual morphological analyzer complements finite-state analyzers where their lexicons are small, and both produce usable analyses for languages unseen in training. Transliteration is almost fully reversible, and converting text to the Common Turkic Alphabet removes most of the additional tokenization cost that large language models incur on Cyrillic- and Arabic-script Turkic languages. Code, models, and documentation are available at: https://github.com/turkic-nlp/turkicnlp.