mlsanityData to 7 Oct 2026

AraBERT: Transformer-based Model for Arabic Language Understanding

arXiv 2003.00104 ↗First posted 28 Feb 2020cs.CLcited by 359

Not on the map. Why?

Abstract (arXiv)

The Arabic language is a morphologically rich language with relatively few resources and a less explored syntax compared to English. Given these limitations, Arabic Natural Language Processing (NLP) tasks like Sentiment Analysis (SA), Named Entity Recognition (NER), and Question Answering (QA), have proven to be very challenging to tackle. Recently, with the surge of transformers based models, language-specific BERT based models have proven to be very efficient at language understanding, provided they are pre-trained on a very large corpus. Such models were able to set new standards and achieve state-of-the-art results for most NLP tasks. In this paper, we pre-trained BERT specifically for the Arabic language in the pursuit of achieving the same success that BERT did for the English language. The performance of AraBERT is compared to multilingual BERT from Google and other state-of-the-art approaches. The results showed that the newly developed AraBERT achieved state-of-the-art performance on most tested Arabic NLP tasks. The pretrained araBERT models are publicly available on https://github.com/aub-mind/arabert hoping to encourage research and applications for Arabic NLP.

Key facts. Cited by 359 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 60; 2022: 57; 2023: 60; 2024: 58; 2025: 69; 2026: 55.

359 papers.

Sort:

By area:

  1. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  2. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  3. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  4. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  5. 2021BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Banglacited by 123
  6. 2021The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Modelscited by 118
  7. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  8. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  9. 2024ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabiccited by 87
  10. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  11. 2022Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methodscited by 75
  12. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 68
38 more
  1. 2021AraT5: Text-to-Text Transformers for Arabic Language Generationcited by 63
  2. 2022Gender Bias in Masked Language Models for Multiple Languagescited by 60
  3. 2022Language Modelling with Pixelscited by 53
  4. 2022ORCA: A Challenging Benchmark for Arabic Language Understandingcited by 46
  5. 2024AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMscited by 46
  6. 2022Token Dropping for Efficient BERT Pretrainingcited by 42
  7. 2021Pre-Training BERT on Arabic Tweets: Practical Considerationscited by 39
  8. 2021QASR: QCRI Aljazeera Speech Resource -- A Large Scale Annotated Arabic Speech Corpuscited by 38
  9. 2021BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translationcited by 35
  10. 2022Speaking Multiple Languages Affects the Moral Bias of Language Modelscited by 34
  11. 2023Lost in Translation: Large Language Models in Non-English Content Analysiscited by 34
  12. 2021NADI 2021: The Second Nuanced Arabic Dialect Identification Shared Taskcited by 32
  13. 2021AMMU : A Survey of Transformer-based Biomedical Pretrained Language Modelscited by 32
  14. 2022Wojood: Nested Arabic Named Entity Corpus and Recognition using BERTcited by 32
  15. 2023Understanding Telecom Language Through Large Language Modelscited by 32
  16. 2023On Evaluating and Mitigating Gender Biases in Multilingual Settingscited by 31
  17. 2023NADI 2023: The Fourth Nuanced Arabic Dialect Identification Shared Taskcited by 29
  18. 2022KinyaBERT: a Morphology-aware Kinyarwanda Language Modelcited by 27
  19. 2021A systematic review of Hate Speech automatic detection using Natural Language Processingcited by 26
  20. 2021Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokenscited by 26
  21. 2022Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languagescited by 25
  22. 2022AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarizationcited by 24
  23. 2022NADI 2022: The Third Nuanced Arabic Dialect Identification Shared Taskcited by 23
  24. 2021ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoderscited by 22
  25. 2022On the Domain Adaptation and Generalization of Pretrained Language Models: A Surveycited by 22
  26. 2023Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigationcited by 22
  27. 2023Dolphin: A Challenging and Diverse Benchmark for Arabic NLGcited by 22
  28. 2024Can GPT-4 Identify Propaganda? Annotation and Detection of Propaganda Spans in News Articlescited by 21
  29. 2021Masader: Metadata Sourcing for Arabic Text and Speech Data Resourcescited by 20
  30. 2024ArabicaQA: A Comprehensive Dataset for Arabic Question Answeringcited by 20
  31. 2024An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Modelscited by 20
  32. 2023Sabi'a: Portuguese Large Language Modelscited by 19
  33. 2023Does Manipulating Tokenization Aid Cross-Lingual Transfer? A Study on POS Tagging for Non-Standardized Languagescited by 19
  34. 2023ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessmentcited by 19
  35. 2023ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Textcited by 19
  36. 2023Large Language Models for Propaganda Span Annotationcited by 19
  37. 2021DziriBERT: a Pre-trained Language Model for the Algerian Dialectcited by 18
  38. 2022ArabGlossBERT: Fine-Tuning BERT on Context-Gloss Pairs for WSDcited by 18

50 most cited of 359 All 359 (JSON)