# FaBERT: Pre-training BERT on Persian Blogs

arXiv: https://arxiv.org/abs/2402.06617 · HTML: https://mlsanity.com/paper/2402.06617/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 9 Feb 2024.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [ParsBERT: Transformer-based Model for Persian Language Understanding](https://mlsanity.com/paper/2005.12515/) (2020)
- [Cramming: Training a Language Model on a Single GPU in One Day](https://mlsanity.com/paper/2212.14034/) (2022)
- [ParsiNLU: A Suite of Language Understanding Challenges for Persian](https://mlsanity.com/paper/2012.06154/) (2020)
- [Pre-Training BERT on Arabic Tweets: Practical Considerations](https://mlsanity.com/paper/2102.10684/) (2021)
- [FarsTail: A Persian Natural Language Inference Dataset](https://mlsanity.com/paper/2009.08820/) (2020)
- [PQuAD: A Persian Question Answering Dataset](https://mlsanity.com/paper/2202.06219/) (2022)
- [DeepSentiPers: Novel Deep Learning Models Trained Over Proposed Augmented Persian Sentiment Corpus](https://mlsanity.com/paper/2004.05328/) (2020)
- [PEYMA: A Tagged Corpus for Persian Named Entities](https://mlsanity.com/paper/1801.09936/) (2018)
- [HmBlogs: A big general Persian corpus](https://mlsanity.com/paper/2111.02362/) (2021)
- [PCoQA: Persian Conversational Question Answering Dataset](https://mlsanity.com/paper/2312.04362/) (2023)
- [The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages](https://mlsanity.com/paper/2310.14557/) (2023)

## Built on by (most cited first, 10 of 10)

- [FaMTEB: Massive Text Embedding Benchmark in Persian Language](https://mlsanity.com/paper/2502.11571/) (2025)
- [TookaBERT: A Step Forward for Persian NLU](https://mlsanity.com/paper/2407.16382/) (2024)
- [On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey](https://mlsanity.com/paper/2507.20783/) (2025)
- [Hakim: Farsi Text Embedding Model](https://mlsanity.com/paper/2505.08435/) (2025)
- [PBBQ: A Persian Bias Benchmark Dataset Curated with Human-AI Collaboration for Large Language Models](https://mlsanity.com/paper/2510.19616/) (2025)
- [PerkwE_COQA: Enhanced Persian Conversational Question Answering by combining contextual keyword extraction with Large Language Models](https://mlsanity.com/paper/2404.05406/) (2024)
- [Formality Style Transfer in Persian](https://mlsanity.com/paper/2406.00867/) (2024)
- [Exploring Subjective Tasks in Farsi: A Survey Analysis and Evaluation of Language Models](https://mlsanity.com/paper/2509.05719/) (2025)
- [PerSoMed: A Large-Scale Balanced Dataset for Persian Social Media Text Classification](https://mlsanity.com/paper/2602.19333/) (2026)
- [IHUBERT: Vector-Based Semantic Deduplication and Domain-Balanced Pretraining for Persian Resources](https://mlsanity.com/paper/2606.20089/) (2026)

## Abstract (arXiv)

We introduce FaBERT, a Persian BERT-base model pre-trained on the HmBlogs corpus, encompassing both informal and formal Persian texts. FaBERT is designed to excel in traditional Natural Language Understanding (NLU) tasks, addressing the intricacies of diverse sentence structures and linguistic styles prevalent in the Persian language. In our comprehensive evaluation of FaBERT on 12 datasets in various downstream tasks, encompassing Sentiment Analysis (SA), Named Entity Recognition (NER), Natural Language Inference (NLI), Question Answering (QA), and Question Paraphrasing (QP), it consistently demonstrated improved performance, all achieved within a compact model size. The findings highlight the importance of utilizing diverse and cleaned corpora, such as HmBlogs, to enhance the performance of language models like BERT in Persian Natural Language Processing (NLP) applications. FaBERT is openly accessible at https://huggingface.co/sbunlp/fabert