mlsanityData to 14 Oct 2026

AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processing

arXiv 2306.06800 ↗First posted 11 Jun 2023cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Developing monolingual large Pre-trained Language Models (PLMs) is shown to be very successful in handling different tasks in Natural Language Processing (NLP). In this work, we present AraMUS, the largest Arabic PLM with 11B parameters trained on 529GB of high-quality Arabic textual data. AraMUS achieves state-of-the-art performances on a diverse set of Arabic classification and generative tasks. Moreover, AraMUS shows impressive few-shot learning abilities compared with the best existing Arabic PLMs.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 42 of its 60 references that are in the map. First posted on arXiv 11 Jun 2023.

Builds on

42 of 60 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
30 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  4. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  5. 2017Mixed Precision Trainingcited by 1,156
  6. 2021Ethical and social risks of harm from Language Modelscited by 966
  7. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  8. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  9. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  10. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 362
  11. 2022UL2: Unifying Language Learning Paradigmscited by 295
  12. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 260
  13. 2020ARBERT & MARBERT: Deep Bidirectional Transformers for Arabiccited by 166
  14. 2021Understanding the Capabilities, Limitations, and Societal Impact of Large Language Modelscited by 160
  15. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  16. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  17. 2021The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Modelscited by 119
  18. 2022ChatGPT: The End of Online Exam Integrity?cited by 88
  19. 2020KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Mediacited by 84
  20. 2022On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Modelcited by 81
  21. 2021AraT5: Text-to-Text Transformers for Arabic Language Generationcited by 63
  22. 2020AraGPT2: Pre-Trained Transformer for Arabic Language Generationcited by 55
  23. 2022ORCA: A Challenging Benchmark for Arabic Language Understandingcited by 47
  24. 2020AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understandingcited by 43
  25. 20161.5 billion words Arabic Corpuscited by 35
  26. 2022AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarizationcited by 24
  27. 2019Arabic natural language processing: An overviewcited by 20
  28. 2022METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signalscited by 20
  29. 2022JASMINE: Arabic GPT Models for Few-Shot Learningcited by 11
  30. 2023A Survey on Arabic Named Entity Recognition: Past, Recent Advances, and Future Trendscited by 8

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 4; 2025: 1.

7 papers.

  1. 2023AceGPT, Localizing Large Language Models in Arabiccited by 75
  2. 2024ALLaM: Large Language Models for Arabic and Englishcited by 55
  3. 2024A Survey of Large Language Models for Arabic Language and its Dialectscited by 13
  4. 2023Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Modelscited by 10
  5. 2025Large Language Models and Arabic Content: A Reviewcited by 4
  6. 2024101 Billion Arabic Words Datasetcited by 2
  7. 2024Arabic Stable LM: Adapting Stable LM 2 1.6B to Arabiccited by 1