# ParsBERT: Transformer-based Model for Persian Language Understanding

arXiv: https://arxiv.org/abs/2005.12515 · HTML: https://mlsanity.com/paper/2005.12515/

## Key facts

- Cited by 93 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 May 2020.

## Built on by (most cited first, 30 of 93)

- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [AMMU : A Survey of Transformer-based Biomedical Pretrained Language Models](https://mlsanity.com/paper/2105.00827/) (2021)
- [The Touch'e23-ValueEval Dataset for Identifying Human Values behind Arguments](https://mlsanity.com/paper/2301.13771/) (2023)
- [GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost](https://mlsanity.com/paper/2211.06993/) (2022)
- [Large-Scale Contextualised Language Modelling for Norwegian](https://mlsanity.com/paper/2104.06546/) (2021)
- [PQuAD: A Persian Question Answering Dataset](https://mlsanity.com/paper/2202.06219/) (2022)
- [gaBERT -- an Irish Language Model](https://mlsanity.com/paper/2107.12930/) (2021)
- [SINA-BERT: A pre-trained Language Model for Analysis of Medical Texts in Persian](https://mlsanity.com/paper/2104.07613/) (2021)
- [FaBERT: Pre-training BERT on Persian Blogs](https://mlsanity.com/paper/2402.06617/) (2024)
- [FaMTEB: Massive Text Embedding Benchmark in Persian Language](https://mlsanity.com/paper/2502.11571/) (2025)
- [ArmanEmo: A Persian Dataset for Text-based Emotion Detection](https://mlsanity.com/paper/2207.11808/) (2022)
- [USTC-NELSLIP at SemEval-2022 Task 11: Gazetteer-Adapted Integration Network for Multilingual Complex Named Entity Recognition](https://mlsanity.com/paper/2203.03216/) (2022)
- [Prose2Poem: The Blessing of Transformers in Translating Prose to Persian Poetry](https://mlsanity.com/paper/2109.14934/) (2021)
- [PersianMind: A Cross-Lingual Persian-English Large Language Model](https://mlsanity.com/paper/2401.06466/) (2024)
- [TookaBERT: A Step Forward for Persian NLU](https://mlsanity.com/paper/2407.16382/) (2024)
- [An Analysis of Social Biases Present in BERT Variants Across Multiple Languages](https://mlsanity.com/paper/2211.14402/) (2022)
- [Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages](https://mlsanity.com/paper/2305.02215/) (2023)
- [PerCQA: Persian Community Question Answering Dataset](https://mlsanity.com/paper/2112.13238/) (2021)
- [naab: A ready-to-use plug-and-play corpus for Farsi](https://mlsanity.com/paper/2208.13486/) (2022)
- [EViT: Privacy-Preserving Image Retrieval via Encrypted Vision Transformer in Cloud Computing](https://mlsanity.com/paper/2208.14657/) (2022)
- [PCoQA: Persian Conversational Question Answering Dataset](https://mlsanity.com/paper/2312.04362/) (2023)
- [evaluating bert and parsbert for analyzing persian advertisement data](https://mlsanity.com/paper/2305.02426/) (2023)
- [Leveraging Online Data to Enhance Medical Knowledge in a Small Persian Language Model](https://mlsanity.com/paper/2505.16000/) (2025)
- [A Novel Context-Aware Multimodal Framework for Persian Sentiment Analysis](https://mlsanity.com/paper/2103.02636/) (2021)
- [BERT-DRE: BERT with Deep Recursive Encoder for Natural Language Sentence Matching](https://mlsanity.com/paper/2111.02188/) (2021)
- [Towards Arabic Sentence Simplification via Classification and Generative Approaches](https://mlsanity.com/paper/2204.09292/) (2022)
- [Adaptive Prompt Learning-based Few-Shot Sentiment Analysis](https://mlsanity.com/paper/2205.07220/) (2022)
- [Opportunities for Persian Digital Humanities Research with Artificial Intelligence Language Models; Case Study: Forough Farrokhzad](https://mlsanity.com/paper/2405.06760/) (2024)
- [PersianMedQA: Evaluating Large Language Models on a Persian-English Bilingual Medical Question Answering Benchmark](https://mlsanity.com/paper/2506.00250/) (2025)

## Abstract (arXiv)

The surge of pre-trained language models has begun a new era in the field of Natural Language Processing (NLP) by allowing us to build powerful language models. Among these models, Transformer-based models such as BERT have become increasingly popular due to their state-of-the-art performance. However, these models are usually focused on English, leaving other languages to multilingual models with limited resources. This paper proposes a monolingual BERT for the Persian language (ParsBERT), which shows its state-of-the-art performance compared to other architectures and multilingual models. Also, since the amount of data available for NLP tasks in Persian is very restricted, a massive dataset for different NLP tasks as well as pre-training the model is composed. ParsBERT obtains higher scores in all datasets, including existing ones as well as composed ones and improves the state-of-the-art performance by outperforming both multilingual BERT and other prior works in Sentiment Analysis, Text Classification and Named Entity Recognition tasks.