mlsanityData to 14 Oct 2026

FaBERT: Pre-training BERT on Persian Blogs

arXiv 2402.06617 ↗First posted 9 Feb 2024cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

We introduce FaBERT, a Persian BERT-base model pre-trained on the HmBlogs corpus, encompassing both informal and formal Persian texts. FaBERT is designed to excel in traditional Natural Language Understanding (NLU) tasks, addressing the intricacies of diverse sentence structures and linguistic styles prevalent in the Persian language. In our comprehensive evaluation of FaBERT on 12 datasets in various downstream tasks, encompassing Sentiment Analysis (SA), Named Entity Recognition (NER), Natural Language Inference (NLI), Question Answering (QA), and Question Paraphrasing (QP), it consistently demonstrated improved performance, all achieved within a compact model size. The findings highlight the importance of utilizing diverse and cleaned corpora, such as HmBlogs, to enhance the performance of language models like BERT in Persian Natural Language Processing (NLP) applications. FaBERT is openly accessible at https://huggingface.co/sbunlp/fabert

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 19 of its 26 references that are in the map. First posted on arXiv 9 Feb 2024.

Builds on

19 of 26 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  5. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  7. 2018CoQA: A Conversational Question Answering Challengecited by 683
  8. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
  9. 2020ParsBERT: Transformer-based Model for Persian Language Understandingcited by 93
  10. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  11. 2020ParsiNLU: A Suite of Language Understanding Challenges for Persiancited by 42
  12. 2021Pre-Training BERT on Arabic Tweets: Practical Considerationscited by 39
7 more
  1. 2020FarsTail: A Persian Natural Language Inference Datasetcited by 23
  2. 2022PQuAD: A Persian Question Answering Datasetcited by 14
  3. 2020DeepSentiPers: Novel Deep Learning Models Trained Over Proposed Augmented Persian Sentiment Corpuscited by 7
  4. 2018PEYMA: A Tagged Corpus for Persian Named Entitiescited by 5
  5. 2021HmBlogs: A big general Persian corpuscited by 5
  6. 2023PCoQA: Persian Conversational Question Answering Datasetcited by 5
  7. 2023The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languagescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 2.

10 papers.

  1. 2025FaMTEB: Massive Text Embedding Benchmark in Persian Languagecited by 10
  2. 2024TookaBERT: A Step Forward for Persian NLUcited by 7
  3. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  4. 2025Hakim: Farsi Text Embedding Modelcited by 2
  5. 2025PBBQ: A Persian Bias Benchmark Dataset Curated with Human-AI Collaboration for Large Language Modelscited by 1
  6. 2024PerkwE_COQA: Enhanced Persian Conversational Question Answering by combining contextual keyword extraction with Large Language Modelscited by 0
  7. 2024Formality Style Transfer in Persiancited by 0
  8. 2025Exploring Subjective Tasks in Farsi: A Survey Analysis and Evaluation of Language Modelscited by 0
  9. 2026PerSoMed: A Large-Scale Balanced Dataset for Persian Social Media Text Classificationcited by 0
  10. 2026IHUBERT: Vector-Based Semantic Deduplication and Domain-Balanced Pretraining for Persian Resourcescited by 0