mlsanityData to 7 Oct 2026

AraGPT2: Pre-Trained Transformer for Arabic Language Generation

arXiv 2012.15520 ↗First posted 31 Dec 2020cs.CLcited by 54

Not on the map. Why?

Abstract (arXiv)

Recently, pre-trained transformer-based architectures have proven to be very efficient at language modeling and understanding, given that they are trained on a large enough corpus. Applications in language generation for Arabic are still lagging in comparison to other NLP advances primarily due to the lack of advanced Arabic language generation models. In this paper, we develop the first advanced Arabic language generation model, AraGPT2, trained from scratch on a large Arabic corpus of internet text and news articles. Our largest model, AraGPT2-mega, has 1.46 billion parameters, which makes it the largest Arabic language model available. The Mega model was evaluated and showed success on different tasks including synthetic news generation, and zero-shot question answering. For text generation, our best model achieves a perplexity of 29.8 on held-out Wikipedia articles. A study conducted with human evaluators showed the significant success of AraGPT2-mega in generating news articles that are difficult to distinguish from articles written by humans. We thus develop and release an automatic discriminator model with a 98% percent accuracy in detecting model-generated text. The models are also publicly available, hoping to encourage new research directions and applications for Arabic NLP.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 9; 2023: 16; 2024: 13; 2025: 10; 2026: 4.

54 papers.

Sort:

By area:

  1. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  2. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  3. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  4. 2024ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabiccited by 87
  5. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 68
  6. 2024ALLaM: Large Language Models for Arabic and Englishcited by 55
  7. 2022Machine Learning Model Sizes and the Parameter Gapcited by 39
  8. 2023Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learningcited by 26
  9. 2023From Text to Source: Results in Detecting Large Language Model-Generated Contentcited by 25
  10. 2022NADI 2022: The Third Nuanced Arabic Dialect Identification Shared Taskcited by 23
  11. 2023Towards a Robust Detection of Language Model Generated Text: Is ChatGPT that Easy to Detect?cited by 23
  12. 2025Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMscited by 20
38 more
  1. 2024An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inferencecited by 18
  2. 2023Prompt Engineering Through the Lens of Optimal Controlcited by 16
  3. 2022Recent Advances in Neural Text Generation: A Task-Agnostic Surveycited by 14
  4. 2024A Survey of Large Language Models for Arabic Language and its Dialectscited by 13
  5. 2022Domain-Specific Text Generation for Machine Translationcited by 12
  6. 2024TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguesecited by 12
  7. 2022JASMINE: Arabic GPT Models for Few-Shot Learningcited by 11
  8. 2023ViDeBERTa: A powerful pre-trained language model for Vietnamesecited by 11
  9. 2021PAGnol: An Extra-Large French Generative Modelcited by 9
  10. 2023NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegiancited by 9
  11. 2025Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gapscited by 9
  12. 2023A Survey on Arabic Named Entity Recognition: Past, Recent Advances, and Future Trendscited by 8
  13. 2024AraPoemBERT: A Pretrained Language Model for Arabic Poetry Analysiscited by 8
  14. 2023Revisiting Entropy Rate Constancy in Textcited by 7
  15. 2023AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processingcited by 7
  16. 2024SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corporacited by 7
  17. 2025The State of Large Language Models for African Languages: Progress and Challengescited by 7
  18. 2023Multilingual Nonce Dependency Treebanks: Understanding how Language Models represent and process syntactic structurecited by 5
  19. 2024Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysiscited by 5
  20. 2024Automated essay scoring in Arabic: a dataset and analysis of a BERT-based systemcited by 5
  21. 2024EgyBERT: A Large Language Model Pretrained on Egyptian Dialect Corporacited by 5
  22. 2025BALSAM: A Platform for Benchmarking Arabic Large Language Modelscited by 5
  23. 2024Demonstration Notebook: Finding the Most Suited In-Context Learning Example from Interactionscited by 4
  24. 2022Data Augmentation using Transformers and Similarity Measures for Improving Arabic Text Classificationcited by 3
  25. 2024Language Modelling Approaches to Adaptive Machine Translationcited by 3
  26. 2022Check-worthy Claim Detection across Topics for Automated Fact-checkingcited by 2
  27. 2023Transformers and Ensemble methods: A solution for Hate Speech Detection in Arabic languagescited by 2
  28. 2025Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfercited by 2
  29. 2025Saudi-Dialect-ALLaM: LoRA Fine-Tuning for Dialectal Arabic Generationcited by 2
  30. 2025A Rhythm-Aware Phrase Insertion for Classical Arabic Poetry Compositioncited by 2
  31. 2026Machine learning and emoji prediction: How much accuracy can MARBERT achieve?cited by 2
  32. 2023Ashaar: Automatic Analysis and Generation of Arabic Poetry Using Deep Learning Approachescited by 1
  33. 2023Detecting Suicidality in Arabic Tweets Using Machine Learning and Deep Learning Techniquescited by 1
  34. 2023Domain Adaptation for Arabic Machine Translation: The Case of Financial Textscited by 1
  35. 2024Arabic Stable LM: Adapting Stable LM 2 1.6B to Arabiccited by 1
  36. 2025Who Wrote This? Identifying Machine vs Human-Generated Text in Hausacited by 1
  37. 2025Towards Inclusive NLP: Assessing Compressed Multilingual Transformers across Diverse Language Benchmarkscited by 1
  38. 2025AraLingBench A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Modelscited by 1

50 most cited of 54 All 54 (JSON)