# DziriBERT: a Pre-trained Language Model for the Algerian Dialect arXiv: https://arxiv.org/abs/2109.12346 ยท HTML: https://mlsanity.com/paper/2109.12346/ ## Key facts - Cited by 18 papers in the map (data through 14 Oct 2026). - Builds on 18 of its 27 references that are in the map. - First posted on arXiv 25 Sep 2021. ## Builds on (most cited first, 18 of 18) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019) - [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013) - [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019) - [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019) - [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018) - [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016) - [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019) - [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020) - [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](https://mlsanity.com/paper/2101.01785/) (2020) - [The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models](https://mlsanity.com/paper/2103.06678/) (2021) - [A Panoramic Survey of Natural Language Processing in the Arab World](https://mlsanity.com/paper/2011.12631/) (2020) - [Pre-Training BERT on Arabic Tweets: Practical Considerations](https://mlsanity.com/paper/2102.10684/) (2021) - [Load What You Need: Smaller Versions of Multilingual BERT](https://mlsanity.com/paper/2010.05609/) (2020) - [Sexism detection: The first corpus in Algerian dialect with a code-switching in Arabic/ French and English](https://mlsanity.com/paper/2104.01443/) (2021) - [The interplay between language similarity and script on a novel multi-layer Algerian dialect corpus](https://mlsanity.com/paper/2105.07400/) (2021) ## Built on by (most cited first, 18 of 18) - [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022) - [SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)](https://mlsanity.com/paper/2304.06845/) (2023) - [SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languages](https://mlsanity.com/paper/2402.08638/) (2024) - [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025) - [A Survey of Large Language Models for Arabic Language and its Dialects](https://mlsanity.com/paper/2410.20238/) (2024) - [Can Character-based Language Models Improve Downstream Task Performance in Low-Resource and Noisy Language Scenarios?](https://mlsanity.com/paper/2110.13658/) (2021) - [A Survey on Arabic Named Entity Recognition: Past, Recent Advances, and Future Trends](https://mlsanity.com/paper/2302.03512/) (2023) - [SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corpora](https://mlsanity.com/paper/2405.06239/) (2024) - [EgyBERT: A Large Language Model Pretrained on Egyptian Dialect Corpora](https://mlsanity.com/paper/2408.03524/) (2024) - [Enriching the NArabizi Treebank: A Multifaceted Approach to Supporting an Under-Resourced Language](https://mlsanity.com/paper/2306.14866/) (2023) - [Large Language Models and Arabic Content: A Review](https://mlsanity.com/paper/2505.08004/) (2025) - [Hate speech detection in algerian dialect using deep learning](https://mlsanity.com/paper/2309.11611/) (2023) - [FASSILA: A Corpus for Algerian Dialect Fake News Detection and Sentiment Analysis](https://mlsanity.com/paper/2411.04604/) (2024) - [MasonTigers at SemEval-2024 Task 1: An Ensemble Approach for Semantic Textual Relatedness](https://mlsanity.com/paper/2403.14990/) (2024) - [chDzDT: Word-level morphology-aware language model for Algerian social media text](https://mlsanity.com/paper/2509.01772/) (2025) - [dziribot: rag based intelligent conversational agent for algerian arabic dialect](https://mlsanity.com/paper/2602.02270/) (2026) - [From FusHa to Folk: Exploring Cross-Lingual Transfer in Arabic Language Models](https://mlsanity.com/paper/2602.09826/) (2026) - [FAD-SA-GRU: Enhancing Hate Speech Detection in Algerian Dialect Through Feature-Augmented Self-Attention GRU Networks](https://mlsanity.com/paper/2607.11279/) (2026) ## Abstract (arXiv) Pre-trained transformers are now the de facto models in Natural Language Processing given their state-of-the-art results in many tasks and languages. However, most of the current models have been trained on languages for which large text resources are already available (such as English, French, Arabic, etc.). Therefore, there are still a number of low-resource languages that need more attention from the community. In this paper, we study the Algerian dialect which has several specificities that make the use of Arabic or multilingual models inappropriate. To address this issue, we collected more than one million Algerian tweets, and pre-trained the first Algerian language model: DziriBERT. When compared with existing models, DziriBERT achieves better results, especially when dealing with the Roman script. The obtained results show that pre-training a dedicated model on a small dataset (150 MB) can outperform existing models that have been trained on much more data (hundreds of GB). Finally, our model is publicly available to the community.