mlsanityData to 13 Oct 2026

JASMINE: Arabic GPT Models for Few-Shot Learning

arXiv 2212.10755 ↗First posted 21 Dec 2022cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Scholarship on generative pretraining (GPT) remains acutely Anglocentric, leaving serious gaps in our understanding of the whole class of autoregressive models. For example, we have little knowledge about the potential of these models and their societal impacts in diverse linguistic and cultural settings. We alleviate this issue for Arabic, a wide collection of languages and dialectal varieties with more than 400 million population, by introducing JASMINE. JASMINE is a suite of powerful Arabic autoregressive Transformer language models ranging in size between 300 million-6.7 billion parameters pretrained on a large and diverse dataset (~ 235 GB of text). We also carefully design and release a comprehensive benchmark for both automated and human evaluation of Arabic autoregressive models, with coverage of potential social biases, harms, and toxicity. Using our novel benchmark, we evaluate JASMINE extensively showing powerful performance intrinsically as well as in few-shot learning on a wide range of NLP tasks. We aim to responsibly release our models and evaluation benchmark with interested researchers, along with code for experimenting with them.

Key facts. Cited by 11 papers in the map (data through 13 Oct 2026). Builds on 54 of its 85 references that are in the map. First posted on arXiv 21 Dec 2022.

Builds on

54 of 85 references.

References · 54

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2022Training language models to follow instructions with human feedbackcited by 15,856
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,785
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,765
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,972
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,693
  11. 2020Learning to summarize from human feedbackcited by 2,575
  12. 2022Emergent Abilities of Large Language Modelscited by 2,499
38 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,139
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,978
  3. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,815
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,468
  5. 2022LaMDA: Language Models for Dialog Applicationscited by 1,219
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,201
  7. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,126
  8. 2021Ethical and social risks of harm from Language Modelscited by 964
  9. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 935
  10. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 918
  11. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  12. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 658
  13. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  14. 2022Improving alignment of dialogue agents via targeted human judgementscited by 529
  15. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 485
  16. 2018Gender Bias in Coreference Resolutioncited by 439
  17. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 361
  18. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 347
  19. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  20. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  21. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  22. 2019Evaluating Gender Bias in Machine Translationcited by 247
  23. 2022Data Contamination: From Memorization to Exploitationcited by 236
  24. 2020ARBERT & MARBERT: Deep Bidirectional Transformers for Arabiccited by 166
  25. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  26. 2021Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Modelscited by 141
  27. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  28. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  29. 2020Learning from Task Descriptionscited by 78
  30. 2021AraT5: Text-to-Text Transformers for Arabic Language Generationcited by 63
  31. 2020AraGPT2: Pre-Trained Transformer for Arabic Language Generationcited by 55
  32. 2020NADI 2020: The First Nuanced Arabic Dialect Identification Shared Taskcited by 47
  33. 2020"You are grounded!": Latent Name Artifacts in Pre-trained Language Modelscited by 39
  34. 20161.5 billion words Arabic Corpuscited by 35
  35. 2022Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Biascited by 33
  36. 2021NADI 2021: The Second Nuanced Arabic Dialect Identification Shared Taskcited by 32
  37. 2020Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Modelscited by 27
  38. 2021Evaluating Gender Bias in Natural Language Inferencecited by 27

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 5; 2026: 1.

11 papers.

  1. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 190
  2. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 69
  3. 2025Palm: A Culturally Inclusive and Linguistically Diverse Dataset for Arabic LLMscited by 15
  4. 2024A Survey of Large Language Models for Arabic Language and its Dialectscited by 13
  5. 2025NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communitiescited by 13
  6. 2025The Landscape of Arabic Large Language Models (ALLMs): A New Era for Arabic Language Technologycited by 12
  7. 2023AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processingcited by 7
  8. 2025Social Bias in Multilingual Language Models: A Surveycited by 2
  9. 2025AraFinNews: Arabic Financial Summarisation with Domain-Adapted LLMscited by 1
  10. 2024On the importance of Data Scale in Pretraining Arabic Language Modelscited by 0
  11. 2026Some Dialects Are More Equal Than Others: Non-Prestigious Arabic Dialectal Bias in LLMscited by 0