mlsanityData to 7 Oct 2026

On the importance of Data Scale in Pretraining Arabic Language Models

arXiv 2401.07760 ↗First posted 15 Jan 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Pretraining monolingual language models have been proven to be vital for performance in Arabic Natural Language Processing (NLP) tasks. In this paper, we conduct a comprehensive study on the role of data in Arabic Pretrained Language Models (PLMs). More precisely, we reassess the performance of a suite of state-of-the-art Arabic PLMs by retraining them on massive-scale, high-quality Arabic corpora. We have significantly improved the performance of the leading Arabic encoder-only BERT-base and encoder-decoder T5-base models on the ALUE and ORCA leaderboards, thereby reporting state-of-the-art results in their respective model categories. In addition, our analysis strongly suggests that pretraining data by far is the primary contributor to performance, surpassing other factors. Our models and source code are publicly available at https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/JABER-PyTorch.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 37 of its 60 references that are in the map. First posted on arXiv 15 Jan 2024.

Builds on

37 of 60 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  11. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  12. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
25 more
  1. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  2. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 359
  3. 2018How Does Batch Normalization Help Optimization?cited by 355
  4. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  5. 2015Semi-supervised Sequence Learningcited by 198
  6. 2020ARBERT & MARBERT: Deep Bidirectional Transformers for Arabiccited by 164
  7. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  8. 2021The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Modelscited by 118
  9. 2020KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Mediacited by 84
  10. 2023AceGPT, Localizing Large Language Models in Arabiccited by 75
  11. 2021AraT5: Text-to-Text Transformers for Arabic Language Generationcited by 63
  12. 2022ORCA: A Challenging Benchmark for Arabic Language Understandingcited by 46
  13. 2020Arabic Dialect Identification in the Wildcited by 45
  14. 2020AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understandingcited by 43
  15. 2018Integrating Stance Detection and Fact Checking in a Unified Corpuscited by 41
  16. 2023GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLPcited by 37
  17. 2020Stance Prediction and Claim Verification: An Arabic Perspectivecited by 25
  18. 2023Dolphin: A Challenging and Diverse Benchmark for Arabic NLGcited by 22
  19. 2020Machine Generation and Detection of Arabic Manipulated and Fake Newscited by 16
  20. 2021Training Multilingual Pre-trained Language Model with Byte-level Subwordscited by 16
  21. 2019AraNet: A Deep Learning Toolkit for Arabic Social Mediacited by 14
  22. 2020Understanding and Detecting Dangerous Speech in Social Mediacited by 12
  23. 2022JASMINE: Arabic GPT Models for Few-Shot Learningcited by 11
  24. 2019NSURL-2019 Shared Task 8: Semantic Question Similarity in Arabiccited by 7
  25. 2021JABER and SABER: Junior and Senior Arabic BERtcited by 3

Built on by

Not cited yet.