# TookaBERT: A Step Forward for Persian NLU

arXiv: https://arxiv.org/abs/2407.16382 · HTML: https://mlsanity.com/paper/2407.16382/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 23 Jul 2024.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [BPE-Dropout: Simple and Effective Subword Regularization](https://mlsanity.com/paper/1910.13267/) (2019)
- [MADLAD-400: A Multilingual And Document-Level Large Audited Dataset](https://mlsanity.com/paper/2309.04662/) (2023)
- [ParsBERT: Transformer-based Model for Persian Language Understanding](https://mlsanity.com/paper/2005.12515/) (2020)
- [XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models](https://mlsanity.com/paper/2301.10472/) (2023)
- [ParsiNLU: A Suite of Language Understanding Challenges for Persian](https://mlsanity.com/paper/2012.06154/) (2020)
- [MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining](https://mlsanity.com/paper/2312.17482/) (2023)
- [FarsTail: A Persian Natural Language Inference Dataset](https://mlsanity.com/paper/2009.08820/) (2020)
- [Boosting Distributed Training Performance of the Unpadded BERT Model](https://mlsanity.com/paper/2208.08124/) (2022)
- [SentiPers: A Sentiment Analysis Corpus for Persian](https://mlsanity.com/paper/1801.07737/) (2018)
- [PQuAD: A Persian Question Answering Dataset](https://mlsanity.com/paper/2202.06219/) (2022)
- [FaBERT: Pre-training BERT on Persian Blogs](https://mlsanity.com/paper/2402.06617/) (2024)
- [ArmanEmo: A Persian Dataset for Text-based Emotion Detection](https://mlsanity.com/paper/2207.11808/) (2022)
- [DeepSentiPers: Novel Deep Learning Models Trained Over Proposed Augmented Persian Sentiment Corpus](https://mlsanity.com/paper/2004.05328/) (2020)
- [PEYMA: A Tagged Corpus for Persian Named Entities](https://mlsanity.com/paper/1801.09936/) (2018)
- [HmBlogs: A big general Persian corpus](https://mlsanity.com/paper/2111.02362/) (2021)

## Built on by (most cited first, 7 of 7)

- [FaMTEB: Massive Text Embedding Benchmark in Persian Language](https://mlsanity.com/paper/2502.11571/) (2025)
- [Matina: A Large-Scale 73B Token Persian Text Corpus](https://mlsanity.com/paper/2502.09188/) (2025)
- [From Millions of Tweets to Actionable Insights: Leveraging LLMs for User Profiling](https://mlsanity.com/paper/2505.06184/) (2025)
- [Advancing Retrieval-Augmented Generation for Persian: Development of Language Models, Comprehensive Benchmarks, and Best Practices for Optimization](https://mlsanity.com/paper/2501.04858/) (2025)
- [Hakim: Farsi Text Embedding Model](https://mlsanity.com/paper/2505.08435/) (2025)
- [PerSoMed: A Large-Scale Balanced Dataset for Persian Social Media Text Classification](https://mlsanity.com/paper/2602.19333/) (2026)
- [IHUBERT: Vector-Based Semantic Deduplication and Domain-Balanced Pretraining for Persian Resources](https://mlsanity.com/paper/2606.20089/) (2026)

## Abstract (arXiv)

The field of natural language processing (NLP) has seen remarkable advancements, thanks to the power of deep learning and foundation models. Language models, and specifically BERT, have been key players in this progress. In this study, we trained and introduced two new BERT models using Persian data. We put our models to the test, comparing them to seven existing models across 14 diverse Persian natural language understanding (NLU) tasks. The results speak for themselves: our larger model outperforms the competition, showing an average improvement of at least +2.8 points. This highlights the effectiveness and potential of our new BERT models for Persian NLU tasks.