# Building Foundations for Natural Language Processing of Historical Turkish: Resources and Models

arXiv: https://arxiv.org/abs/2501.04828 · HTML: https://mlsanity.com/paper/2501.04828/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 52 references that are in the map.
- First posted on arXiv 8 Jan 2025.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [UL2: Unifying Language Learning Paradigms](https://mlsanity.com/paper/2205.05131/) (2022)
- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [Named Entity Recognition and Classification on Historical Documents: A Survey](https://mlsanity.com/paper/2109.11406/) (2021)
- [Parsing as Pretraining](https://mlsanity.com/paper/2002.01685/) (2020)
- [TURNA: A Turkish Encoder-Decoder Language Model for Enhanced Understanding and Generation](https://mlsanity.com/paper/2401.14373/) (2024)
- [A Hybrid Persian Sentiment Analysis Framework: Integrating Dependency Grammar Based Rules and Deep Neural Networks](https://mlsanity.com/paper/1909.13568/) (2019)
- [Is POS Tagging Necessary or Even Helpful for Neural Dependency Parsing?](https://mlsanity.com/paper/2003.03204/) (2020)
- [Applying Occam's Razor to Transformer-Based Dependency Parsing: What Works, What Doesn't, and What is Really Necessary](https://mlsanity.com/paper/2010.12699/) (2020)
- [Resources for Turkish Dependency Parsing: Introducing the BOUN Treebank and the BoAT Annotation Tool](https://mlsanity.com/paper/2002.10416/) (2020)
- [RGAT: A Deeper Look into Syntactic Dependency Information for Coreference Resolution](https://mlsanity.com/paper/2309.04977/) (2023)
- [Dependency Annotation of Ottoman Turkish with Multilingual BERT](https://mlsanity.com/paper/2402.14743/) (2024)
- [A Hybrid Approach to Dependency Parsing: Combining Rules and Morphology with Deep Learning](https://mlsanity.com/paper/2002.10116/) (2020)
- [Joint POS Tagging and Dependency Parsing with Transition-based Neural Networks](https://mlsanity.com/paper/1704.07616/) (2017)
- [A multi-level multi-label text classification dataset of 19th century Ottoman and Russian literary and critical texts](https://mlsanity.com/paper/2407.15136/) (2024)

## Abstract (arXiv)

This paper introduces foundational resources and models for natural language processing (NLP) of historical Turkish, a domain that has remained underexplored in computational linguistics. We present the first named entity recognition (NER) dataset, HisTR, and the first Universal Dependencies treebank, OTA-BOUN, for a historical form of the Turkish language along with transformer-based models trained using these datasets for named entity recognition, dependency parsing, and part-of-speech tagging tasks. Furthermore, we introduce the Ottoman Text Corpus (OTC), a clean corpus of transliterated historical Turkish texts that spans a wide range of historical periods. Our experimental results demonstrate prominent improvements in the computational analysis of historical Turkish, achieving strong performance on tasks that require understanding of historical linguistic structures -- specifically, 90.29% F1 in named entity recognition, 73.79% LAS for dependency parsing, and 94.98% F1 for part-of-speech tagging. They also highlight existing challenges, such as domain adaptation and language variations between time periods. All the resources and models presented are available at https://hf.co/bucolin to serve as a benchmark for future progress in historical Turkish NLP.