# MizanQA: Benchmarking Large Language Models on Moroccan Legal Question Answering

arXiv: https://arxiv.org/abs/2508.16357 · HTML: https://mlsanity.com/paper/2508.16357/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 11 references that are in the map.
- First posted on arXiv 22 Aug 2025.

## Builds on (most cited first, 3 of 3)

- [LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain](https://mlsanity.com/paper/2408.10343/) (2024)
- [MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks](https://mlsanity.com/paper/2505.03427/) (2025)
- [Can Large Language Models Predict the Outcome of Judicial Decisions?](https://mlsanity.com/paper/2501.09768/) (2025)

## Built on by (most cited first, 5 of 5)

- [Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps](https://mlsanity.com/paper/2510.13430/) (2025)
- [IslamicLegalBench: Evaluating LLMs Knowledge and Reasoning of Islamic Law Across 1,200 Years of Islamic Pluralist Legal Traditions](https://mlsanity.com/paper/2602.21226/) (2026)
- [Fanar-Sadiq: A Multi-Agent Architecture for Grounded Islamic QA](https://mlsanity.com/paper/2603.08501/) (2026)
- [Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation](https://mlsanity.com/paper/2604.03395/) (2026)
- [QuranicMMLU: A Cognitively-Aware Benchmark for Evaluating Generative AI Solutions on Quranic Linguistic Knowledge](https://mlsanity.com/paper/2609.22038/) (2026)

## Abstract (arXiv)

The rapid advancement of large language models (LLMs) has significantly propelled progress in natural language processing (NLP). However, their effectiveness in specialized, low-resource domains-such as Arabic legal contexts-remains limited. This paper introduces MizanQA (pronounced Mizan, meaning "scale" in Arabic, a universal symbol of justice), a benchmark designed to evaluate LLMs on Moroccan legal question answering (QA) tasks, characterised by rich linguistic and legal complexity. The dataset draws on Modern Standard Arabic, Islamic Maliki jurisprudence, Moroccan customary law, and French legal influences. Comprising over 1,700 multiple-choice questions, including multi-answer formats, MizanQA captures the nuances of authentic legal reasoning. Benchmarking experiments with multilingual and Arabic-focused LLMs reveal substantial performance gaps, highlighting the need for tailored evaluation metrics and culturally grounded, domain-specific LLM development.