# Pre-Training BERT on Arabic Tweets: Practical Considerations

arXiv: https://arxiv.org/abs/2102.10684 · HTML: https://mlsanity.com/paper/2102.10684/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 30 references that are in the map.
- First posted on arXiv 21 Feb 2021.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020)
- [SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)](https://mlsanity.com/paper/2006.07235/) (2020)
- [A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages](https://mlsanity.com/paper/2006.06202/) (2020)
- [Multilingual is not enough: BERT for Finnish](https://mlsanity.com/paper/1912.07076/) (2019)
- [BERTje: A Dutch BERT Model](https://mlsanity.com/paper/1912.09582/) (2019)
- [RobBERT: a Dutch RoBERTa-based Language Model](https://mlsanity.com/paper/2001.06286/) (2020)
- [KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media](https://mlsanity.com/paper/2007.13184/) (2020)
- [Arabic Dialect Identification in the Wild](https://mlsanity.com/paper/2005.06557/) (2020)
- [1.5 billion words Arabic Corpus](https://mlsanity.com/paper/1611.04033/) (2016)
- [Arabic Offensive Language on Twitter: Analysis and Experiments](https://mlsanity.com/paper/2004.02192/) (2020)
- [Multi-Dialect Arabic BERT for Country-Level Dialect Identification](https://mlsanity.com/paper/2007.05612/) (2020)
- [AnchiBERT: A Pre-Trained Model for Ancient ChineseLanguage Understanding and Generation](https://mlsanity.com/paper/2009.11473/) (2020)

## Built on by (most cited first, 30 of 39)

- [Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://mlsanity.com/paper/2308.16149/) (2023)
- [AraT5: Text-to-Text Transformers for Arabic Language Generation](https://mlsanity.com/paper/2109.12068/) (2021)
- [Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigation](https://mlsanity.com/paper/2305.14734/) (2023)
- [ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text](https://mlsanity.com/paper/2311.03179/) (2023)
- [DziriBERT: a Pre-trained Language Model for the Algerian Dialect](https://mlsanity.com/paper/2109.12346/) (2021)
- [ArabGlossBERT: Fine-Tuning BERT on Context-Gloss Pairs for WSD](https://mlsanity.com/paper/2205.09685/) (2022)
- [Evaluating Various Tokenizers for Arabic Text Classification](https://mlsanity.com/paper/2106.07540/) (2021)
- [Morphosyntactic Tagging with Pre-trained Language Models for Arabic and its Dialects](https://mlsanity.com/paper/2110.06852/) (2021)
- [Emojis as Anchors to Detect Arabic Offensive Language and Hate Speech](https://mlsanity.com/paper/2201.06723/) (2022)
- [D-CALM: A Dynamic Clustering-based Active Learning Approach for Mitigating Bias](https://mlsanity.com/paper/2305.17013/) (2023)
- [The Landscape of Arabic Large Language Models (ALLMs): A New Era for Arabic Language Technology](https://mlsanity.com/paper/2506.01340/) (2025)
- [FaBERT: Pre-training BERT on Persian Blogs](https://mlsanity.com/paper/2402.06617/) (2024)
- [MemeIntel: Explainable Detection of Propagandistic and Hateful Memes](https://mlsanity.com/paper/2502.16612/) (2025)
- [LU-BZU at SemEval-2021 Task 2: Word2Vec and Lemma2Vec performance in Arabic Word-in-Context disambiguation](https://mlsanity.com/paper/2104.08110/) (2021)
- [Propaganda to Hate: A Multimodal Analysis of Arabic Memes with Multi-Agent LLMs](https://mlsanity.com/paper/2409.07246/) (2024)
- [User-Centric Gender Rewriting](https://mlsanity.com/paper/2205.02211/) (2022)
- [AraLegal-BERT: A pretrained language model for Arabic Legal text](https://mlsanity.com/paper/2210.08284/) (2022)
- [ArabianGPT: Native Arabic GPT-based Large Language Model](https://mlsanity.com/paper/2402.15313/) (2024)
- [Offensive Hebrew Corpus and Detection using BERT](https://mlsanity.com/paper/2309.02724/) (2023)
- [WojoodNER 2023: The First Arabic Named Entity Recognition Shared Task](https://mlsanity.com/paper/2310.16153/) (2023)
- [ExaASC: A General Target-Based Stance Detection Corpus in Arabic Language](https://mlsanity.com/paper/2204.13979/) (2022)
- [ArMeme: Propagandistic Content in Arabic Memes](https://mlsanity.com/paper/2406.03916/) (2024)
- [Large Language Models and Arabic Content: A Review](https://mlsanity.com/paper/2505.08004/) (2025)
- [ArCovidVac: Analyzing Arabic Tweets About COVID-19 Vaccination](https://mlsanity.com/paper/2201.06496/) (2022)
- [Optimizing RAG Pipelines for Arabic: A Systematic Analysis of Core Components](https://mlsanity.com/paper/2506.06339/) (2025)
- [TCE at Qur'an QA 2022: Arabic Language Question Answering Over Holy Qur'an Using a Post-Processed Ensemble of BERT-based Models](https://mlsanity.com/paper/2206.01550/) (2022)
- [A Benchmark Study of Contrastive Learning for Arabic Social Meaning](https://mlsanity.com/paper/2210.12314/) (2022)
- [ThatiAR: Subjectivity Detection in Arabic News Sentences](https://mlsanity.com/paper/2406.05559/) (2024)
- [CVPD at QIAS 2025 Shared Task: An Efficient Encoder-Based Approach for Islamic Inheritance Reasoning](https://mlsanity.com/paper/2509.00457/) (2025)
- [Post-hoc analysis of Arabic transformer models](https://mlsanity.com/paper/2210.09990/) (2022)

## Abstract (arXiv)

Pretraining Bidirectional Encoder Representations from Transformers (BERT) for downstream NLP tasks is a non-trival task. We pretrained 5 BERT models that differ in the size of their training sets, mixture of formal and informal Arabic, and linguistic preprocessing. All are intended to support Arabic dialects and social media. The experiments highlight the centrality of data diversity and the efficacy of linguistically aware segmentation. They also highlight that more data or more training step do not necessitate better models. Our new models achieve new state-of-the-art results on several downstream tasks. The resulting models are released to the community under the name QARiB.