# EgyBERT: A Large Language Model Pretrained on Egyptian Dialect Corpora

arXiv: https://arxiv.org/abs/2408.03524 · HTML: https://mlsanity.com/paper/2408.03524/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 43 references that are in the map.
- First posted on arXiv 7 Aug 2024.

## Builds on (most cited first, 14 of 14)

- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020)
- [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](https://mlsanity.com/paper/2101.01785/) (2020)
- [The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models](https://mlsanity.com/paper/2103.06678/) (2021)
- [KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media](https://mlsanity.com/paper/2007.13184/) (2020)
- [AraT5: Text-to-Text Transformers for Arabic Language Generation](https://mlsanity.com/paper/2109.12068/) (2021)
- [AraGPT2: Pre-Trained Transformer for Arabic Language Generation](https://mlsanity.com/paper/2012.15520/) (2020)
- [ORCA: A Challenging Benchmark for Arabic Language Understanding](https://mlsanity.com/paper/2212.10758/) (2022)
- [1.5 billion words Arabic Corpus](https://mlsanity.com/paper/1611.04033/) (2016)
- [AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarization](https://mlsanity.com/paper/2203.10945/) (2022)
- [DziriBERT: a Pre-trained Language Model for the Algerian Dialect](https://mlsanity.com/paper/2109.12346/) (2021)
- [AraPoemBERT: A Pretrained Language Model for Arabic Poetry Analysis](https://mlsanity.com/paper/2403.12392/) (2024)
- [AraLegal-BERT: A pretrained language model for Arabic Legal text](https://mlsanity.com/paper/2210.08284/) (2022)
- [SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corpora](https://mlsanity.com/paper/2405.06239/) (2024)

## Built on by (most cited first, 5 of 5)

- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [A Survey of Large Language Models for Arabic Language and its Dialects](https://mlsanity.com/paper/2410.20238/) (2024)
- [Nile-Chat: Egyptian Language Models for Arabic and Latin Scripts](https://mlsanity.com/paper/2507.04569/) (2025)
- [Large Language Models and Arabic Content: A Review](https://mlsanity.com/paper/2505.08004/) (2025)
- [From FusHa to Folk: Exploring Cross-Lingual Transfer in Arabic Language Models](https://mlsanity.com/paper/2602.09826/) (2026)

## Abstract (arXiv)

This study presents EgyBERT, an Arabic language model pretrained on 10.4 GB of Egyptian dialectal texts. We evaluated EgyBERT's performance by comparing it with five other multidialect Arabic language models across 10 evaluation datasets. EgyBERT achieved the highest average F1-score of 84.25% and an accuracy of 87.33%, significantly outperforming all other comparative models, with MARBERTv2 as the second best model achieving an F1-score 83.68% and an accuracy 87.19%. Additionally, we introduce two novel Egyptian dialectal corpora: the Egyptian Tweets Corpus (ETC), containing over 34.33 million tweets (24.89 million sentences) amounting to 2.5 GB of text, and the Egyptian Forums Corpus (EFC), comprising over 44.42 million sentences (7.9 GB of text) collected from various Egyptian online forums. Both corpora are used in pretraining the new model, and they are the largest Egyptian dialectal corpora to date reported in the literature. Furthermore, this is the first study to evaluate the performance of various language models on Egyptian dialect datasets, revealing significant differences in performance that highlight the need for more dialect-specific models. The results confirm the effectiveness of EgyBERT model in processing and analyzing Arabic text expressed in Egyptian dialect, surpassing other language models included in the study. EgyBERT model is publicly available on urlhttps://huggingface.co/faisalq/EgyBERT.