# Developing and Evaluating Tiny to Medium-Sized Turkish BERT Models

arXiv: https://arxiv.org/abs/2307.14134 · HTML: https://mlsanity.com/paper/2307.14134/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 19 references that are in the map.
- First posted on arXiv 26 Jul 2023.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](https://mlsanity.com/paper/2201.06642/) (2022)
- [Small and Practical BERT Models for Sequence Labeling](https://mlsanity.com/paper/1909.00100/) (2019)
- [FinEst BERT and CroSloEngual BERT: less is more in multilingual models](https://mlsanity.com/paper/2006.07890/) (2020)
- [High Quality ELMo Embeddings for Seven Less-Resourced Languages](https://mlsanity.com/paper/1911.10049/) (2019)

## Built on by (most cited first, 8 of 8)

- [Introducing cosmosGPT: Monolingual Training for Turkish Language Models](https://mlsanity.com/paper/2404.17336/) (2024)
- [TurkishBERTweet: Fast and Reliable Large Language Model for Social Media Analysis](https://mlsanity.com/paper/2311.18063/) (2023)
- [VBART: The Turkish LLM](https://mlsanity.com/paper/2403.01308/) (2024)
- [Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domain](https://mlsanity.com/paper/2601.16018/) (2026)
- [MoganBert-TR: A Turkish Encoder Foundation Model Trained from Scratch with a CLM-to-MLM Curriculum](https://mlsanity.com/paper/2608.25768/) (2026)
- [TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish](https://mlsanity.com/paper/2512.23065/) (2025)
- [T"urkcce Dil Modellerinin Performans Karcsilacstirmasi Performance Comparison of Turkish Language Models](https://mlsanity.com/paper/2404.17010/) (2024)
- [TurkEmbed: Turkish Embedding Model on NLI & STS Tasks](https://mlsanity.com/paper/2511.08376/) (2025)

## Abstract (arXiv)

This study introduces and evaluates tiny, mini, small, and medium-sized uncased Turkish BERT models, aiming to bridge the research gap in less-resourced languages. We trained these models on a diverse dataset encompassing over 75GB of text from multiple sources and tested them on several tasks, including mask prediction, sentiment analysis, news classification, and, zero-shot classification. Despite their smaller size, our models exhibited robust performance, including zero-shot task, while ensuring computational efficiency and faster execution times. Our findings provide valuable insights into the development and application of smaller language models, especially in the context of the Turkish language.