mlsanityData to 7 Oct 2026

Pre-Training BERT on Arabic Tweets: Practical Considerations

arXiv 2102.10684 ↗First posted 21 Feb 2021cs.CL cs.AIcited by 39

Not on the map. Why?

Abstract (arXiv)

Pretraining Bidirectional Encoder Representations from Transformers (BERT) for downstream NLP tasks is a non-trival task. We pretrained 5 BERT models that differ in the size of their training sets, mixture of formal and informal Arabic, and linguistic preprocessing. All are intended to support Arabic dialects and social media. The experiments highlight the centrality of data diversity and the efficacy of linguistically aware segmentation. They also highlight that more data or more training step do not necessitate better models. Our new models achieve new state-of-the-art results on several downstream tasks. The resulting models are released to the community under the name QARiB.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 16 of its 30 references that are in the map. First posted on arXiv 21 Feb 2021.

Builds on

16 of 30 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  4. 2019CamemBERT: a Tasty French Language Modelcited by 368
  5. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 359
  6. 2020SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)cited by 138
  7. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  8. 2019Multilingual is not enough: BERT for Finnishcited by 118
  9. 2019BERTje: A Dutch BERT Modelcited by 109
  10. 2020RobBERT: a Dutch RoBERTa-based Language Modelcited by 88
  11. 2020KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Mediacited by 84
  12. 2020Arabic Dialect Identification in the Wildcited by 45
4 more
  1. 20161.5 billion words Arabic Corpuscited by 35
  2. 2020Arabic Offensive Language on Twitter: Analysis and Experimentscited by 28
  3. 2020Multi-Dialect Arabic BERT for Country-Level Dialect Identificationcited by 15
  4. 2020AnchiBERT: A Pre-Trained Model for Ancient ChineseLanguage Understanding and Generationcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 10; 2023: 8; 2024: 8; 2025: 7; 2026: 1.

39 papers.

Sort:

By area:

  1. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 68
  2. 2021AraT5: Text-to-Text Transformers for Arabic Language Generationcited by 63
  3. 2023Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigationcited by 22
  4. 2023ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Textcited by 19
  5. 2021DziriBERT: a Pre-trained Language Model for the Algerian Dialectcited by 18
  6. 2022ArabGlossBERT: Fine-Tuning BERT on Context-Gloss Pairs for WSDcited by 18
  7. 2021Evaluating Various Tokenizers for Arabic Text Classificationcited by 14
  8. 2021Morphosyntactic Tagging with Pre-trained Language Models for Arabic and its Dialectscited by 14
  9. 2022Emojis as Anchors to Detect Arabic Offensive Language and Hate Speechcited by 14
  10. 2023D-CALM: A Dynamic Clustering-based Active Learning Approach for Mitigating Biascited by 12
  11. 2025The Landscape of Arabic Large Language Models (ALLMs): A New Era for Arabic Language Technologycited by 12
  12. 2024FaBERT: Pre-training BERT on Persian Blogscited by 10
27 more
  1. 2021LU-BZU at SemEval-2021 Task 2: Word2Vec and Lemma2Vec performance in Arabic Word-in-Context disambiguationcited by 9
  2. 2024Propaganda to Hate: A Multimodal Analysis of Arabic Memes with Multi-Agent LLMscited by 8
  3. 2025MemeIntel: Explainable Detection of Propagandistic and Hateful Memescited by 8
  4. 2022User-Centric Gender Rewritingcited by 7
  5. 2022AraLegal-BERT: A pretrained language model for Arabic Legal textcited by 7
  6. 2024ArabianGPT: Native Arabic GPT-based Large Language Modelcited by 7
  7. 2023Offensive Hebrew Corpus and Detection using BERTcited by 6
  8. 2023WojoodNER 2023: The First Arabic Named Entity Recognition Shared Taskcited by 6
  9. 2022ExaASC: A General Target-Based Stance Detection Corpus in Arabic Languagecited by 5
  10. 2024ArMeme: Propagandistic Content in Arabic Memescited by 5
  11. 2025Large Language Models and Arabic Content: A Reviewcited by 4
  12. 2022ArCovidVac: Analyzing Arabic Tweets About COVID-19 Vaccinationcited by 3
  13. 2025Optimizing RAG Pipelines for Arabic: A Systematic Analysis of Core Componentscited by 3
  14. 2022TCE at Qur'an QA 2022: Arabic Language Question Answering Over Holy Qur'an Using a Post-Processed Ensemble of BERT-based Modelscited by 2
  15. 2022A Benchmark Study of Contrastive Learning for Arabic Social Meaningcited by 2
  16. 2024ThatiAR: Subjectivity Detection in Arabic News Sentencescited by 2
  17. 2025CVPD at QIAS 2025 Shared Task: An Efficient Encoder-Based Approach for Islamic Inheritance Reasoningcited by 2
  18. 2022Post-hoc analysis of Arabic transformer modelscited by 1
  19. 2023Nexus at ArAIEval Shared Task: Fine-Tuning Arabic Language Models for Propaganda and Disinformation Detectioncited by 1
  20. 2024DarijaBanking: A New Resource for Overcoming Language Barriers in Banking Intent Detection for Moroccan Arabic Speakerscited by 1
  21. 2024Improving Arabic Multi-Label Emotion Classification using Stacked Embeddings and Hybrid Loss Functioncited by 1
  22. 2025SHAMI-MT: A Syrian Arabic Dialect to Modern Standard Arabic Bidirectional Machine Translation Systemcited by 1
  23. 2026Adapting Reinforcement Learning with Chain-of-Thought Supervision for Explainable Detection of Hateful and Propagandistic Memescited by 1
  24. 2022Interpreting Arabic Transformer Modelscited by 0
  25. 2023Masakhane-Afrisenti at SemEval-2023 Task 12: Sentiment Analysis using Afro-centric Language Models and Adapters for Low-resource African Languagescited by 0
  26. 2024Arabic Tweet Act: A Weighted Ensemble Pre-Trained Transformer Model for Classifying Arabic Speech Acts on Twittercited by 0
  27. 2025Arabic Hate Speech Identification and Masking in Social Media using Deep Learning Models and Pre-trained Models Fine-tuningcited by 0