# IslamicMMLU: A Benchmark for Evaluating LLMs on Islamic Knowledge

arXiv: https://arxiv.org/abs/2603.23750 · HTML: https://mlsanity.com/paper/2603.23750/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 29 references that are in the map.
- First posted on arXiv 24 Mar 2026.

## Builds on (most cited first, 12 of 12)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](https://mlsanity.com/paper/2305.14456/) (2023)
- [ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic](https://mlsanity.com/paper/2402.12840/) (2024)
- [AceGPT, Localizing Large Language Models in Arabic](https://mlsanity.com/paper/2309.12053/) (2023)
- [Fanar: An Arabic-Centric Multimodal Generative AI Platform](https://mlsanity.com/paper/2501.13944/) (2025)
- [PalmX 2025: The First Shared Task on Benchmarking LLMs on Arabic and Islamic Culture](https://mlsanity.com/paper/2509.02550/) (2025)
- [Sacred or Synthetic? Evaluating LLM Reliability and Abstention for Religious Questions](https://mlsanity.com/paper/2508.08287/) (2025)
- [IslamicLegalBench: Evaluating LLMs Knowledge and Reasoning of Islamic Law Across 1,200 Years of Islamic Pluralist Legal Traditions](https://mlsanity.com/paper/2602.21226/) (2026)
- ["The Prophet said so!": On Exploring Hadith Presence on Arabic Social Media](https://mlsanity.com/paper/2412.20581/) (2024)
- [Fabricating Holiness: Characterizing Religious Misinformation Circulators on Arabic Social Media](https://mlsanity.com/paper/2508.07845/) (2025)

## Built on by (most cited first, 7 of 7)

- [HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answering](https://mlsanity.com/paper/2607.20219/) (2026)
- [HalluTruthQA-4K: A Fine-Grained Corpus and Annotation Process for Arabic Hallucination Detection and Truth Verification](https://mlsanity.com/paper/2608.03966/) (2026)
- [Islamic Large Language Models: From Knowledge Acquisition to Trustworthy and Hallucination-Resistant AI](https://mlsanity.com/paper/2606.16629/) (2026)
- [AI and Authenticity in Islamic Research: A Critical Evaluation of Generative AI Reliability, Hallucination, and Source Fidelity in Quranic, Hadith, and Fiqh Knowledge](https://mlsanity.com/paper/2607.28237/) (2026)
- [IslamicTurathBench: A Multi-Task, Multi-Discipline Benchmark for Evaluating Large Language Models on the Islamic Scholarly Tradition (turath)](https://mlsanity.com/paper/2608.04703/) (2026)
- [Right Frame, Wrong Rule: Cultural Cues Expose the Financial Knowledge Gap They Were Meant to Close](https://mlsanity.com/paper/2609.00999/) (2026)
- [QuranicMMLU: A Cognitively-Aware Benchmark for Evaluating Generative AI Solutions on Quranic Linguistic Knowledge](https://mlsanity.com/paper/2609.22038/) (2026)

## Abstract (arXiv)

Large language models are increasingly consulted for Islamic knowledge, yet no comprehensive benchmark evaluates their performance across core Islamic disciplines. We introduce IslamicMMLU, a benchmark of 10,013 multiple-choice questions spanning three tracks: Quran (2,013 questions), Hadith (4,000 questions), and Fiqh (jurisprudence, 4,000 questions). Each track is formed of multiple types of questions to examine LLMs capabilities handling different aspects of Islamic knowledge. The benchmark is used to create the IslamicMMLU public leaderboard for evaluating LLMs, and we initially evaluate 26 LLMs, where their averaged accuracy across the three tracks varied between 39.8% to 93.8% (by Gemini 3 Flash). The Quran track shows the widest span (99.3% to 32.4%), while the Fiqh track includes a novel madhab (Islamic school of jurisprudence) bias detection task revealing variable school-of-thought preferences across models. Arabic-specific models show mixed results, but they all underperform compared to frontier models. The evaluation code and leaderboard are made publicly available.