# IHUBERT: Vector-Based Semantic Deduplication and Domain-Balanced Pretraining for Persian Resources

arXiv: https://arxiv.org/abs/2606.20089 · HTML: https://mlsanity.com/paper/2606.20089/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 18 Jun 2026.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [ParsBERT: Transformer-based Model for Persian Language Understanding](https://mlsanity.com/paper/2005.12515/) (2020)
- [GottBERT: a pure German Language Model](https://mlsanity.com/paper/2012.02110/) (2020)
- [ParsiNLU: A Suite of Language Understanding Challenges for Persian](https://mlsanity.com/paper/2012.06154/) (2020)
- [FarsTail: A Persian Natural Language Inference Dataset](https://mlsanity.com/paper/2009.08820/) (2020)
- [KR-BERT: A Small-Scale Korean-Specific Language Model](https://mlsanity.com/paper/2008.03979/) (2020)
- [PQuAD: A Persian Question Answering Dataset](https://mlsanity.com/paper/2202.06219/) (2022)
- [SINA-BERT: A pre-trained Language Model for Analysis of Medical Texts in Persian](https://mlsanity.com/paper/2104.07613/) (2021)
- [FaBERT: Pre-training BERT on Persian Blogs](https://mlsanity.com/paper/2402.06617/) (2024)
- [DeepSentiPers: Novel Deep Learning Models Trained Over Proposed Augmented Persian Sentiment Corpus](https://mlsanity.com/paper/2004.05328/) (2020)
- [TookaBERT: A Step Forward for Persian NLU](https://mlsanity.com/paper/2407.16382/) (2024)
- [PCoQA: Persian Conversational Question Answering Dataset](https://mlsanity.com/paper/2312.04362/) (2023)
- [PERLEX: A Bilingual Persian-English Gold Dataset for Relation Extraction](https://mlsanity.com/paper/2005.06588/) (2020)
- [FarSSiBERT: A Novel Transformer-based Model for Semantic Similarity Measurement of Persian Social Networks Informal Texts](https://mlsanity.com/paper/2407.19173/) (2024)
- [Hakim: Farsi Text Embedding Model](https://mlsanity.com/paper/2505.08435/) (2025)

## Abstract (arXiv)

Persian pretrained language models (PLMs) are still limited by the scarcity of large-scale, high-quality pretraining corpora and by insufficient evaluation beyond standard classification and NER tasks. We present IHUBERT, a monolingual Persian PLM trained from scratch with the RoBERTa-base encoder (125M parameters) on a 45 GB curated subset of the Sepahr-Danesh collection (about 7-8B tokens). To improve corpus quality and reduce redundancy, we employ a multi-stage preprocessing pipeline that includes normalization, exact and near-duplicate removal, anonymization, and vector-database-based semantic deduplication for distribution balancing control across domains and registers. We additionally train a 139k-vocabulary BPE tokenizer on the full pretraining corpus to better capture Persian morphology and orthographic variation. IHUBERT is evaluated on seven Persian NLU benchmarks covering NER, sentiment analysis, topic classification, NLI, extractive question answering, and relation extraction, using task-standard metrics (entity-level F1, Macro-F1, EM/F1). IHUBERT achieves its strongest gains on extractive QA, ranking first on both PQuAD (F1 88.3542) and ParsiNLU-RC (F1 49.0987), and attains the best result on FarsTail (Macro-F1 0.8350). On NER and topic classification, it remains competitive (e.g., 0.8308 F1 on ParsTwiNER; 0.7953 Macro-F1 on DigiMag), while relation extraction remains the main remaining gap (0.6684 Macro-F1 on PERLEX). A controlled tokenizer ablation on the IHUBERT pretraining corpus shows that BPE yields slightly lower subword fragmentation than WordPiece at matched vocabulary size, supporting our tokenization design. Overall, IHUBERT advances Persian language modeling through semantically curated large-scale pretraining and broad evaluation across both classification and comprehension-oriented tasks.