# AraGPT2: Pre-Trained Transformer for Arabic Language Generation

arXiv: https://arxiv.org/abs/2012.15520 · HTML: https://mlsanity.com/paper/2012.15520/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2020.

## Built on by (most cited first, 30 of 55)

- [Compute Trends Across Three Eras of Machine Learning](https://mlsanity.com/paper/2202.05924/) (2022)
- [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](https://mlsanity.com/paper/2305.14456/) (2023)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic](https://mlsanity.com/paper/2402.12840/) (2024)
- [Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://mlsanity.com/paper/2308.16149/) (2023)
- [ALLaM: Large Language Models for Arabic and English](https://mlsanity.com/paper/2407.15390/) (2024)
- [Machine Learning Model Sizes and the Parameter Gap](https://mlsanity.com/paper/2207.02852/) (2022)
- [Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning](https://mlsanity.com/paper/2301.09626/) (2023)
- [From Text to Source: Results in Detecting Large Language Model-Generated Content](https://mlsanity.com/paper/2309.13322/) (2023)
- [NADI 2022: The Third Nuanced Arabic Dialect Identification Shared Task](https://mlsanity.com/paper/2210.09582/) (2022)
- [Towards a Robust Detection of Language Model Generated Text: Is ChatGPT that Easy to Detect?](https://mlsanity.com/paper/2306.05871/) (2023)
- [Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs](https://mlsanity.com/paper/2502.18795/) (2025)
- [An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inference](https://mlsanity.com/paper/2402.10712/) (2024)
- [Prompt Engineering Through the Lens of Optimal Control](https://mlsanity.com/paper/2310.14201/) (2023)
- [Recent Advances in Neural Text Generation: A Task-Agnostic Survey](https://mlsanity.com/paper/2203.03047/) (2022)
- [TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese](https://mlsanity.com/paper/2401.16640/) (2024)
- [A Survey of Large Language Models for Arabic Language and its Dialects](https://mlsanity.com/paper/2410.20238/) (2024)
- [Domain-Specific Text Generation for Machine Translation](https://mlsanity.com/paper/2208.05909/) (2022)
- [JASMINE: Arabic GPT Models for Few-Shot Learning](https://mlsanity.com/paper/2212.10755/) (2022)
- [ViDeBERTa: A powerful pre-trained language model for Vietnamese](https://mlsanity.com/paper/2301.10439/) (2023)
- [PAGnol: An Extra-Large French Generative Model](https://mlsanity.com/paper/2110.08554/) (2021)
- [NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian](https://mlsanity.com/paper/2312.01314/) (2023)
- [Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps](https://mlsanity.com/paper/2510.13430/) (2025)
- [A Survey on Arabic Named Entity Recognition: Past, Recent Advances, and Future Trends](https://mlsanity.com/paper/2302.03512/) (2023)
- [AraPoemBERT: A Pretrained Language Model for Arabic Poetry Analysis](https://mlsanity.com/paper/2403.12392/) (2024)
- [The State of Large Language Models for African Languages: Progress and Challenges](https://mlsanity.com/paper/2506.02280/) (2025)
- [Revisiting Entropy Rate Constancy in Text](https://mlsanity.com/paper/2305.12084/) (2023)
- [AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processing](https://mlsanity.com/paper/2306.06800/) (2023)
- [SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corpora](https://mlsanity.com/paper/2405.06239/) (2024)
- [Multilingual Nonce Dependency Treebanks: Understanding how Language Models represent and process syntactic structure](https://mlsanity.com/paper/2311.07497/) (2023)

## Abstract (arXiv)

Recently, pre-trained transformer-based architectures have proven to be very efficient at language modeling and understanding, given that they are trained on a large enough corpus. Applications in language generation for Arabic are still lagging in comparison to other NLP advances primarily due to the lack of advanced Arabic language generation models. In this paper, we develop the first advanced Arabic language generation model, AraGPT2, trained from scratch on a large Arabic corpus of internet text and news articles. Our largest model, AraGPT2-mega, has 1.46 billion parameters, which makes it the largest Arabic language model available. The Mega model was evaluated and showed success on different tasks including synthetic news generation, and zero-shot question answering. For text generation, our best model achieves a perplexity of 29.8 on held-out Wikipedia articles. A study conducted with human evaluators showed the significant success of AraGPT2-mega in generating news articles that are difficult to distinguish from articles written by humans. We thus develop and release an automatic discriminator model with a 98% percent accuracy in detecting model-generated text. The models are also publicly available, hoping to encourage new research directions and applications for Arabic NLP.