mlsanityData to 7 Oct 2026

The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models

arXiv 2103.06678 ↗First posted 11 Mar 2021cs.CLcited by 118

Not on the map. Why?

Abstract (arXiv)

In this paper, we explore the effects of language variants, data sizes, and fine-tuning task types in Arabic pre-trained language models. To do so, we build three pre-trained language models across three variants of Arabic: Modern Standard Arabic (MSA), dialectal Arabic, and classical Arabic, in addition to a fourth language model which is pre-trained on a mix of the three. We also examine the importance of pre-training data size by building additional models that are pre-trained on a scaled-down set of the MSA variant. We compare our different models to each other, as well as to eight publicly available models by fine-tuning them on five NLP tasks spanning 12 datasets. Our results suggest that the variant proximity of pre-training data to fine-tuning data is more important than the pre-training data size. We exploit this insight in defining an optimized system selection model for the studied tasks.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Builds on 25 of its 63 references that are in the map. First posted on arXiv 11 Mar 2021.

Builds on

25 of 63 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  4. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  5. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  6. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 359
  7. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  8. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  9. 2019SemEval-2017 Task 4: Sentiment Analysis in Twittercited by 209
  10. 2020ARBERT & MARBERT: Deep Bidirectional Transformers for Arabiccited by 164
  11. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  12. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
13 more
  1. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  2. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  3. 2020KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Mediacited by 84
  4. 2020NADI 2020: The First Nuanced Arabic Dialect Identification Shared Taskcited by 47
  5. 2020Arabic Dialect Identification in the Wildcited by 45
  6. 20161.5 billion words Arabic Corpuscited by 35
  7. 2020Do Neural Language Models Show Preferences for Syntactic Formalisms?cited by 30
  8. 2018Arap-Tweet: A Large Multi-Dialect Twitter Corpus for Gender, Age and Language Variety Identificationcited by 23
  9. 2020On the importance of pre-training data volume for compact language modelscited by 21
  10. 2020Multi-Dialect Arabic BERT for Country-Level Dialect Identificationcited by 15
  11. 2019Learning meters of Arabic and English poems with Recurrent Neural Networks: a step forward for language understanding and synthesiscited by 13
  12. 2019ArSentD-LEV: A Multi-Topic Corpus for Target-based Sentiment Analysis in Arabic Levantine Tweetscited by 11
  13. 2017WERd: Using Social Text Spelling Variants for Evaluating Dialectal Speech Recognitioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 15; 2023: 17; 2024: 30; 2025: 29; 2026: 21.

118 papers.

Sort:

By area:

  1. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  2. 2024ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabiccited by 87
  3. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 68
  4. 2021AraT5: Text-to-Text Transformers for Arabic Language Generationcited by 63
  5. 2022ORCA: A Challenging Benchmark for Arabic Language Understandingcited by 46
  6. 2024AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMscited by 46
  7. 2023SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)cited by 41
  8. 2024Natural Language Processing for Dialects of a Language: A Surveycited by 40
  9. 2022AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarizationcited by 24
  10. 2023Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigationcited by 22
  11. 2021DziriBERT: a Pre-trained Language Model for the Algerian Dialectcited by 18
  12. 2022ArabGlossBERT: Fine-Tuning BERT on Context-Gloss Pairs for WSDcited by 18
38 more
  1. 2024CIDAR: Culturally Relevant Instruction Dataset For Arabiccited by 17
  2. 2023TARJAMAT: Evaluation of Bard and ChatGPT on Machine Translation of Ten Arabic Varietiescited by 16
  3. 2025Palm: A Culturally Inclusive and Linguistically Diverse Dataset for Arabic LLMscited by 15
  4. 2021Morphosyntactic Tagging with Pre-trained Language Models for Arabic and its Dialectscited by 14
  5. 2024A Survey of Large Language Models for Arabic Language and its Dialectscited by 13
  6. 2024AraFinNLP 2024: The First Arabic Financial NLP Shared Taskcited by 12
  7. 2025The Landscape of Arabic Large Language Models (ALLMs): A New Era for Arabic Language Technologycited by 12
  8. 2024ArAIEval Shared Task: Propagandistic Techniques Detection in Unimodal and Multimodal Arabic Contentcited by 11
  9. 2024Swan and ArabicMTEB: Dialect-Aware, Arabic-Centric, Cross-Lingual, and Cross-Cultural Embedding Models and Benchmarkscited by 10
  10. 2021LU-BZU at SemEval-2021 Task 2: Word2Vec and Lemma2Vec performance in Arabic Word-in-Context disambiguationcited by 9
  11. 2025Enhancing Text Editing for Grammatical Error Correction: Arabic as a Case Studycited by 9
  12. 2023GreekBART: The First Pretrained Greek Sequence-to-Sequence Modelcited by 8
  13. 2023ArBanking77: Intent Detection Neural Model and a New Dataset in Modern and Dialectical Arabiccited by 8
  14. 2024AraPoemBERT: A Pretrained Language Model for Arabic Poetry Analysiscited by 8
  15. 2025A Large and Balanced Corpus for Fine-grained Arabic Readability Assessmentcited by 8
  16. 2022User-Centric Gender Rewritingcited by 7
  17. 2023AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processingcited by 7
  18. 2024ArabianGPT: Native Arabic GPT-based Large Language Modelcited by 7
  19. 2024SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corporacited by 7
  20. 2025JEEM: Vision-Language Understanding in Four Arabic Dialectscited by 7
  21. 2023WojoodNER 2023: The First Arabic Named Entity Recognition Shared Taskcited by 6
  22. 2022Benchmarking Evaluation Metrics for Code-Switching Automatic Speech Recognitioncited by 5
  23. 2024Cross-Lingual Transfer from Related Languages: Treating Low-Resource Maltese as Multilingual Code-Switchingcited by 5
  24. 2024EgyBERT: A Large Language Model Pretrained on Egyptian Dialect Corporacited by 5
  25. 2024From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languagescited by 5
  26. 2025Enhancing Arabic Automated Essay Scoring with Synthetic Data and Error Injectioncited by 5
  27. 2021Supporting Undotted Arabic with Pre-trained Language Modelscited by 4
  28. 2022DTW at Qur'an QA 2022: Utilising Transfer Learning with Transformers for Question Answering in a Low-resource Domaincited by 4
  29. 2023Leveraging Domain Adaptation and Data Augmentation to Improve Qur'anic IR in English and Arabiccited by 4
  30. 2024Arabic Text Diacritization In The Age Of Transfer Learning: Token Classification Is All You Needcited by 4
  31. 2024MentalQA: An Annotated Arabic Corpus for Questions and Answers of Mental Healthcarecited by 4
  32. 2025Multi-stage Training of Bilingual Islamic LLM for Neural Passage Retrievalcited by 4
  33. 2021JABER and SABER: Junior and Senior Arabic BERtcited by 3
  34. 2023HausaNLP at SemEval-2023 Task 12: Leveraging African Low Resource TweetData for Sentiment Analysiscited by 3
  35. 2024Arabic Synonym BERT-based Adversarial Examples for Text Classificationcited by 3
  36. 2024Guidelines for Fine-grained Sentence-level Arabic Readability Annotationcited by 3
  37. 2025Abjad AI at NADI 2025: CATT-Whisper: Multimodal Diacritic Restoration Using Text and Speech Representationscited by 3
  38. 2025CARMA: Comprehensive Automatically-annotated Reddit Mental Health Dataset for Arabiccited by 3

50 most cited of 118 All 118 (JSON)