mlsanityData to 7 Oct 2026

ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic

arXiv 2402.12840 ↗First posted 20 Feb 2024cs.CLcited by 87

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The focus of language model evaluation has transitioned towards reasoning and knowledge-intensive tasks, driven by advancements in pretraining large models. While state-of-the-art models are partially trained on large Arabic texts, evaluating their performance in Arabic remains challenging due to the limited availability of relevant datasets. To bridge this gap, we present datasetname, the first multi-task language understanding benchmark for the Arabic language, sourced from school exams across diverse educational levels in different countries spanning North Africa, the Levant, and the Gulf regions. Our data comprises 40 tasks and 14,575 multiple-choice questions in Modern Standard Arabic (MSA) and is carefully constructed by collaborating with native speakers in the region. Our comprehensive evaluations of 35 models reveal substantial room for improvement, particularly among the best open-source models. Notably, BLOOMZ, mT0, LLaMA2, and Falcon struggle to achieve a score of 50%, while even the top-performing Arabic-centric model only achieves a score of 62.3%.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 40 of its 58 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

40 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU7 Oct

Other references · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2021Program Synthesis with Large Language Modelscited by 3,509
  10. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  11. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  12. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
27 more
  1. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  2. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  3. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  4. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  5. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 359
  6. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  7. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  8. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  9. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  10. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  11. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  12. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 167
  13. 2020ARBERT & MARBERT: Deep Bidirectional Transformers for Arabiccited by 164
  14. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  15. 2021The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Modelscited by 118
  16. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 100
  17. 2023AceGPT, Localizing Large Language Models in Arabiccited by 75
  18. 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
  19. 2023Language models are not naysayers: An analysis of language models on negation benchmarkscited by 73
  20. 2021AraT5: Text-to-Text Transformers for Arabic Language Generationcited by 63
  21. 2020EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answeringcited by 60
  22. 2020AraGPT2: Pre-Trained Transformer for Arabic Language Generationcited by 54
  23. 2019Neural Arabic Question Answeringcited by 48
  24. 2022ORCA: A Challenging Benchmark for Arabic Language Understandingcited by 46
  25. 2020AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understandingcited by 43
  26. 2022AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarizationcited by 24
  27. 2023Dolphin: A Challenging and Diverse Benchmark for Arabic NLGcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 38; 2026: 25.

87 papers.

Sort:

By area:

  1. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 78
  2. 2024ALLaM: Large Language Models for Arabic and Englishcited by 55
  3. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languagescited by 55
  4. 2024AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMscited by 46
  5. 2024IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provincescited by 38
  6. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 33
  7. 2024CaLMQA: Exploring culturally specific long-form question answering across 23 languagescited by 31
  8. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  9. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  10. 2024BertaQA: How Much Do Language Models Know About Local Culture?cited by 23
  11. 2024Unveiling Linguistic Regions in Large Language Modelscited by 22
  12. 2024CAMEL-Bench: A Comprehensive Arabic LMM Benchmarkcited by 17
38 more
  1. 2025Command R7B Arabic: A Small, Enterprise Focused, Multilingual, and Culturally Aware Arabic LLMcited by 16
  2. 2024ProverbEval: Exploring LLM Evaluation Challenges for Low-resource Language Understandingcited by 15
  3. 2024Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialectcited by 13
  4. 2025MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languagescited by 13
  5. 2024BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanismcited by 12
  6. 2024ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Modelscited by 11
  7. 2024CamelEval: Advancing Culturally Aligned Arabic Language Models and Benchmarkscited by 10
  8. 2025Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Abilitycited by 10
  9. 2025Sacred or Synthetic? Evaluating LLM Reliability and Abstention for Religious Questionscited by 10
  10. 2025Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gapscited by 9
  11. 2024Cultural Adaptation of Menus: A Fine-Grained Approachcited by 8
  12. 2025AIN: The Arabic INclusive Large Multimodal Modelcited by 8
  13. 2025TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languagescited by 8
  14. 2025A Large and Balanced Corpus for Fine-grained Arabic Readability Assessmentcited by 8
  15. 2025HKCanto-Eval: A Benchmark for Evaluating Cantonese Language Understanding and Cultural Comprehension in LLMscited by 8
  16. 2025MultiHoax: A Dataset of Multi-hop False-Premise Questionscited by 8
  17. 2024Alignment at Pre-training! Towards Native Alignment for Arabic LLMscited by 7
  18. 2025JEEM: Vision-Language Understanding in Four Arabic Dialectscited by 7
  19. 2025Nile-Chat: Egyptian Language Models for Arabic and Latin Scriptscited by 7
  20. 2025LLM Alignment for the Arabs: A Homogenous Culture or Diverse Ones?cited by 6
  21. 2025BnMMLU: Measuring Massive Multitask Language Understanding in Bengalicited by 6
  22. 2025From Guidelines to Practice: A New Paradigm for Arabic Language Model Evaluationcited by 6
  23. 2025SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhalacited by 6
  24. 2025DialectalArabicMMLU: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Modelscited by 6
  25. 2026IslamicMMLU: A Benchmark for Evaluating LLMs on Islamic Knowledgecited by 6
  26. 2025KazMMLU: Evaluating Language Models on Kazakh, Russian, and Regional Knowledge of Kazakhstancited by 5
  27. 2025Enhancing Arabic Automated Essay Scoring with Synthetic Data and Error Injectioncited by 5
  28. 2025Emergent Abilities of Large Language Models under Continued Pretraining for Language Adaptationcited by 5
  29. 2025BALSAM: A Platform for Benchmarking Arabic Large Language Modelscited by 5
  30. 2024Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesiacited by 4
  31. 2024Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansioncited by 4
  32. 2025IRLBench: A Multi-modal, Culturally Grounded, Parallel Irish-English Benchmark for Open-Ended LLM Reasoning Evaluationcited by 4
  33. 2025MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMscited by 4
  34. 2024Guidelines for Fine-grained Sentence-level Arabic Readability Annotationcited by 3
  35. 2025Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answeringcited by 3
  36. 2026Apples to Apples? Towards Comparable Crosslingual Language Model Evaluationcited by 3
  37. 2024Language Adaptation on a Tight Academic Compute Budget: Tokenizer Swapping Works and Pure bfloat16 Is Enoughcited by 2
  38. 2025Language and Planning in Robotic Navigation: A Multilingual Evaluation of State-of-the-Art Modelscited by 2

50 most cited of 87 All 87 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Mind the Gap: A Review of Arabic Post-Training Datasets and Their Limitations
  2. 2024The Qiyas Benchmark: Measuring ChatGPT Mathematical and Language Understanding in Arabic
  3. 2025UI-Level Evaluation of ALLaM 34B: Measuring an Arabic-Centric LLM via HUMAIN Chat
  4. 2024AraSTEM: A Native Arabic Multiple Choice Question Benchmark for Evaluating LLMs Knowledge In STEM Subjects
  5. 2025Beyond MCQ: An Open-Ended Arabic Cultural QA Benchmark with Dialect Variants
  6. 2026Cultural Benchmarking of LLMs in Standard and Dialectal Arabic Dialogues
  7. 2025AraTable: Benchmarking LLMs' Reasoning and Understanding of Arabic Tabular Data
  8. 2025AraReasoner: Evaluating Reasoning-Based LLMs for Arabic NLP