mlsanityData to 14 Oct 2026

GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP

arXiv 2305.14976 ↗First posted 24 May 2023cs.CL cs.LGcited by 37

Not on the map. Why?

Abstract (arXiv)

ChatGPT's emergence heralds a transformative phase in NLP, particularly demonstrated through its excellent performance on many English benchmarks. However, the model's efficacy across diverse linguistic contexts remains largely uncharted territory. This work aims to bridge this knowledge gap, with a primary focus on assessing ChatGPT's capabilities on Arabic languages and dialectal varieties. Our comprehensive study conducts a large-scale automated and human evaluation of ChatGPT, encompassing 44 distinct language understanding and generation tasks on over 60 different datasets. To our knowledge, this marks the first extensive performance analysis of ChatGPT's deployment in Arabic NLP. Our findings indicate that, despite its remarkable performance in English, ChatGPT is consistently surpassed by smaller models that have undergone finetuning on Arabic. We further undertake a meticulous comparison of ChatGPT and GPT-4's Modern Standard Arabic (MSA) and Dialectal Arabic (DA), unveiling the relative shortcomings of both models in handling Arabic dialects compared to MSA. Although we further explore and confirm the utility of employing GPT-4 as a potential alternative for human evaluation, our work adds to a growing body of research underscoring the limitations of ChatGPT.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Builds on 57 of its 100 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

57 of 100 references.

References · 57

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  11. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  12. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
38 more
  1. 2023LIMA: Less Is More for Alignmentcited by 1,100
  2. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  3. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  4. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  5. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  6. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  7. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 652
  8. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619
  9. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  10. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  11. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
  12. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  13. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  14. 2023Can Large Language Models Transform Computational Social Science?cited by 250
  15. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 232
  16. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 211
  17. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  18. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 176
  19. 2020ARBERT & MARBERT: Deep Bidirectional Transformers for Arabiccited by 166
  20. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 165
  21. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 160
  22. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  23. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  24. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  25. 2023Is ChatGPT a Good Sentiment Analyzer? A Preliminary Studycited by 96
  26. 2021AraT5: Text-to-Text Transformers for Arabic Language Generationcited by 63
  27. 2023Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinationscited by 63
  28. 2023How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Taskscited by 58
  29. 2023Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remarkcited by 54
  30. 2020NADI 2020: The First Nuanced Arabic Dialect Identification Shared Taskcited by 47
  31. 2022ORCA: A Challenging Benchmark for Arabic Language Understandingcited by 47
  32. 2023ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbotscited by 47
  33. 2020Arabic Dialect Identification in the Wildcited by 45
  34. 2018Integrating Stance Detection and Fact Checking in a Unified Corpuscited by 41
  35. 2023In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPTcited by 37
  36. 2023How to Design Translation Prompts for ChatGPT: An Empirical Studycited by 31
  37. 2020Stance Prediction and Claim Verification: An Arabic Perspectivecited by 25
  38. 2019Arabic Text Diacritization Using Deep Neural Networkscited by 18

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 21; 2025: 8; 2026: 3.

37 papers.

Sort:

By area:

  1. 2024Investigating Cultural Alignment of Large Language Modelscited by 143
  2. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 129
  3. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 123
  4. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 85
  5. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 70
  6. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  7. 2024AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMscited by 46
  8. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 43
  9. 2024How Does Quantization Affect Multilingual LLMs?cited by 43
  10. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  11. 2023LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarkingcited by 20
  12. 2024Mix Data or Merge Models? Optimizing for Diverse Multi-Task Learningcited by 17
25 more
  1. 2023TARJAMAT: Evaluation of Bard and ChatGPT on Machine Translation of Ten Arabic Varietiescited by 16
  2. 2024AraTrust: An Evaluation of Trustworthiness for LLMs in Arabiccited by 16
  3. 2024Benchmarking Large Language Models for Persian: A Preliminary Study Focusing on ChatGPTcited by 11
  4. 2023Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Modelscited by 10
  5. 2024DetoxLLM: A Framework for Detoxification with Explanationscited by 10
  6. 2024Native vs Non-Native Language Prompting: A Comparative Analysiscited by 9
  7. 2025Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Taskscited by 9
  8. 2025From Guidelines to Practice: A New Paradigm for Arabic Language Model Evaluationcited by 6
  9. 2024Generalists vs. Specialists: Evaluating Large Language Models for Urducited by 5
  10. 2025A Survey of Code-switched Arabic NLP: Progress, Challenges, and Future Directionscited by 5
  11. 2025CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignmentcited by 3
  12. 2024ThatiAR: Subjectivity Detection in Arabic News Sentencescited by 2
  13. 2025Culture Matters in Toxic Language Detection in Persiancited by 2
  14. 2023Domain Adaptation for Arabic Machine Translation: The Case of Financial Textscited by 1
  15. 2024Effective Self-Mining of In-Context Examples for Unsupervised Machine Translation with LLMscited by 1
  16. 2025Survey of NLU Benchmarks Diagnosing Linguistic Phenomena: Why not Standardize Diagnostics Benchmarks?cited by 1
  17. 2023Exploring the Capabilities of ChatGPT in Ancient Chinese Translation and Person Name Recognitioncited by 0
  18. 2024On the importance of Data Scale in Pretraining Arabic Language Modelscited by 0
  19. 2024A Survey on the Real Power of ChatGPTcited by 0
  20. 2024A Novel Psychometrics-Based Approach to Developing Professional Competency Benchmark for Large Language Modelscited by 0
  21. 2025A Comparative Benchmark of a Moroccan Darija Toxicity Detection Model (Typica.ai) and Major LLM-Based Moderation APIs (OpenAI, Mistral, Anthropic)cited by 0
  22. 2025Exploring Data and Parameter Efficient Strategies for Arabic Dialect Identificationscited by 0
  23. 2026MIRAGE: Auditing Anti-Muslim Bias in Frontier LLMs Across Reasoning, Agentic, and Time-Coupled Conditionscited by 0
  24. 2026Evaluation of Small Language Models for Arabic Language Processingcited by 0
  25. 2026Evaluation of Adversarial Robustness in Arabic Language Modelscited by 0