mlsanityData to 7 Oct 2026

TookaBERT: A Step Forward for Persian NLU

arXiv 2407.16382 ↗First posted 23 Jul 2024cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

The field of natural language processing (NLP) has seen remarkable advancements, thanks to the power of deep learning and foundation models. Language models, and specifically BERT, have been key players in this progress. In this study, we trained and introduced two new BERT models using Persian data. We put our models to the test, comparing them to seven existing models across 14 diverse Persian natural language understanding (NLU) tasks. The results speak for themselves: our larger model outperforms the competition, showing an average improvement of at least +2.8 points. This highlights the effectiveness and potential of our new BERT models for Persian NLU tasks.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 26 of its 32 references that are in the map. First posted on arXiv 23 Jul 2024.

Builds on

26 of 32 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  12. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
14 more
  1. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  2. 2020ParsBERT: Transformer-based Model for Persian Language Understandingcited by 92
  3. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  4. 2020ParsiNLU: A Suite of Language Understanding Challenges for Persiancited by 42
  5. 2023MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretrainingcited by 33
  6. 2020FarsTail: A Persian Natural Language Inference Datasetcited by 23
  7. 2022Boosting Distributed Training Performance of the Unpadded BERT Modelcited by 15
  8. 2018SentiPers: A Sentiment Analysis Corpus for Persiancited by 14
  9. 2022PQuAD: A Persian Question Answering Datasetcited by 14
  10. 2024FaBERT: Pre-training BERT on Persian Blogscited by 10
  11. 2022ArmanEmo: A Persian Dataset for Text-based Emotion Detectioncited by 9
  12. 2020DeepSentiPers: Novel Deep Learning Models Trained Over Proposed Augmented Persian Sentiment Corpuscited by 7
  13. 2018PEYMA: A Tagged Corpus for Persian Named Entitiescited by 5
  14. 2021HmBlogs: A big general Persian corpuscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025FaMTEB: Massive Text Embedding Benchmark in Persian Languagecited by 10
  2. 2025Matina: A Large-Scale 73B Token Persian Text Corpuscited by 3
  3. 2025From Millions of Tweets to Actionable Insights: Leveraging LLMs for User Profilingcited by 3
  4. 2025Advancing Retrieval-Augmented Generation for Persian: Development of Language Models, Comprehensive Benchmarks, and Best Practices for Optimizationcited by 2
  5. 2025Hakim: Farsi Text Embedding Modelcited by 2
  6. 2026PerSoMed: A Large-Scale Balanced Dataset for Persian Social Media Text Classificationcited by 0
  7. 2026IHUBERT: Vector-Based Semantic Deduplication and Domain-Balanced Pretraining for Persian Resourcescited by 0