# AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understanding

arXiv: https://arxiv.org/abs/2012.15516 · HTML: https://mlsanity.com/paper/2012.15516/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2020.

## Built on by (most cited first, 30 of 43)

- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic](https://mlsanity.com/paper/2402.12840/) (2024)
- [Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://mlsanity.com/paper/2308.16149/) (2023)
- [ORCA: A Challenging Benchmark for Arabic Language Understanding](https://mlsanity.com/paper/2212.10758/) (2022)
- [NADI 2021: The Second Nuanced Arabic Dialect Identification Shared Task](https://mlsanity.com/paper/2103.08466/) (2021)
- [NADI 2023: The Fourth Nuanced Arabic Dialect Identification Shared Task](https://mlsanity.com/paper/2310.16117/) (2023)
- [Towards a Robust Detection of Language Model Generated Text: Is ChatGPT that Easy to Detect?](https://mlsanity.com/paper/2306.05871/) (2023)
- [Masader: Metadata Sourcing for Arabic Text and Speech Data Resources](https://mlsanity.com/paper/2110.06744/) (2021)
- [ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text](https://mlsanity.com/paper/2311.03179/) (2023)
- [Large Language Models for Propaganda Span Annotation](https://mlsanity.com/paper/2311.09812/) (2023)
- [A Survey of Large Language Models for Arabic Language and its Dialects](https://mlsanity.com/paper/2410.20238/) (2024)
- [A Survey on Arabic Named Entity Recognition: Past, Recent Advances, and Future Trends](https://mlsanity.com/paper/2302.03512/) (2023)
- [AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processing](https://mlsanity.com/paper/2306.06800/) (2023)
- [ArabianGPT: Native Arabic GPT-based Large Language Model](https://mlsanity.com/paper/2402.15313/) (2024)
- [Optimized Quran Passage Retrieval Using an Expanded QA Dataset and Fine-Tuned Language Models](https://mlsanity.com/paper/2412.11431/) (2024)
- [AraBERT and Farasa Segmentation Based Approach For Sarcasm and Sentiment Detection in Arabic Tweets](https://mlsanity.com/paper/2103.01679/) (2021)
- [DTW at Qur'an QA 2022: Utilising Transfer Learning with Transformers for Question Answering in a Low-resource Domain](https://mlsanity.com/paper/2205.06025/) (2022)
- [Data Augmentation using Transformers and Similarity Measures for Improving Arabic Text Classification](https://mlsanity.com/paper/2212.13939/) (2022)
- [CLEANANERCorp: Identifying and Correcting Incorrect Labels in the ANERcorp Dataset](https://mlsanity.com/paper/2408.12362/) (2024)
- [The Arabic AI Fingerprint: Stylometric Analysis and Detection of Large Language Models Text](https://mlsanity.com/paper/2505.23276/) (2025)
- [Transformers and Ensemble methods: A solution for Hate Speech Detection in Arabic languages](https://mlsanity.com/paper/2303.09823/) (2023)
- [Two-Stage Quranic QA via Ensemble Retrieval and Instruction-Tuned Answer Extraction](https://mlsanity.com/paper/2508.06971/) (2025)
- [Dialect Identification in Nuanced Arabic Tweets Using Farasa Segmentation and AraBERT](https://mlsanity.com/paper/2102.09749/) (2021)
- [Post-hoc analysis of Arabic transformer models](https://mlsanity.com/paper/2210.09990/) (2022)
- [Detecting Suicidality in Arabic Tweets Using Machine Learning and Deep Learning Techniques](https://mlsanity.com/paper/2309.00246/) (2023)
- [Ensemble of pre-trained language models and data augmentation for hate speech detection from Arabic tweets](https://mlsanity.com/paper/2407.02448/) (2024)
- [Advancing Arabic Reverse Dictionary Systems: A Transformer-Based Approach with Dataset Construction Guidelines](https://mlsanity.com/paper/2504.21475/) (2025)
- [Enhanced Arabic Text Retrieval with Attentive Relevance Scoring](https://mlsanity.com/paper/2507.23404/) (2025)
- [AHaSIS: Shared Task on Sentiment Analysis for Arabic Dialects](https://mlsanity.com/paper/2511.13335/) (2025)

## Abstract (arXiv)

Advances in English language representation enabled a more sample-efficient pre-training task by Efficiently Learning an Encoder that Classifies Token Replacements Accurately (ELECTRA). Which, instead of training a model to recover masked tokens, it trains a discriminator model to distinguish true input tokens from corrupted tokens that were replaced by a generator network. On the other hand, current Arabic language representation approaches rely only on pretraining via masked language modeling. In this paper, we develop an Arabic language representation model, which we name AraELECTRA. Our model is pretrained using the replaced token detection objective on large Arabic text corpora. We evaluate our model on multiple Arabic NLP tasks, including reading comprehension, sentiment analysis, and named-entity recognition and we show that AraELECTRA outperforms current state-of-the-art Arabic language representation models, given the same pretraining data and with even a smaller model size.