# AL-QASIDA: Analyzing LLM Quality and Accuracy Systematically in Dialectal Arabic

arXiv: https://arxiv.org/abs/2412.04193 · HTML: https://mlsanity.com/paper/2412.04193/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 44 references that are in the map.
- First posted on arXiv 5 Dec 2024.

## Builds on (most cited first, 19 of 19)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.16039/) (2023)
- [Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning](https://mlsanity.com/paper/2402.06619/) (2024)
- [The Generative AI Paradox: "What It Can Create, It May Not Understand"](https://mlsanity.com/paper/2311.00059/) (2023)
- [Do Multilingual Language Models Think Better in English?](https://mlsanity.com/paper/2308.01223/) (2023)
- [AceGPT, Localizing Large Language Models in Arabic](https://mlsanity.com/paper/2309.12053/) (2023)
- [Understanding and Mitigating Language Confusion in LLMs](https://mlsanity.com/paper/2406.20052/) (2024)
- [AraT5: Text-to-Text Transformers for Arabic Language Generation](https://mlsanity.com/paper/2109.12068/) (2021)
- [DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languages](https://mlsanity.com/paper/2403.11009/) (2024)
- [Natural Language Processing for Dialects of a Language: A Survey](https://mlsanity.com/paper/2401.05632/) (2024)
- [NADI 2023: The Fourth Nuanced Arabic Dialect Identification Shared Task](https://mlsanity.com/paper/2310.16117/) (2023)
- [ALDi: Quantifying the Arabic Level of Dialectness of Text](https://mlsanity.com/paper/2310.13747/) (2023)
- [NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Task](https://mlsanity.com/paper/2407.04910/) (2024)
- [Dolphin: A Challenging and Diverse Benchmark for Arabic NLG](https://mlsanity.com/paper/2305.14989/) (2023)
- [Towards Responsible Natural Language Annotation for the Varieties of Arabic](https://mlsanity.com/paper/2203.09597/) (2022)

## Built on by (most cited first, 11 of 11)

- [Command R7B Arabic: A Small, Enterprise Focused, Multilingual, and Culturally Aware Arabic LLM](https://mlsanity.com/paper/2503.14603/) (2025)
- [Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps](https://mlsanity.com/paper/2510.13430/) (2025)
- [Nile-Chat: Egyptian Language Models for Arabic and Latin Scripts](https://mlsanity.com/paper/2507.04569/) (2025)
- [Revisiting Common Assumptions about Arabic Dialects in NLP](https://mlsanity.com/paper/2505.21816/) (2025)
- [From Words to Proverbs: Evaluating LLMs Linguistic and Cultural Competence in Saudi Dialects with Absher](https://mlsanity.com/paper/2507.10216/) (2025)
- [IDIOLEX: Unified and Continuous Representations for Idiolectal and Stylistic Variation](https://mlsanity.com/paper/2604.04704/) (2026)
- [Aladdin-FTI @ AMIYA Three Wishes for Arabic NLP: Fidelity, Diglossia, and Multidialectal Generation](https://mlsanity.com/paper/2602.16290/) (2026)
- [Multicultural Spyfall: Assessing LLMs through Dynamic Multilingual Social Deduction Game](https://mlsanity.com/paper/2601.09017/) (2026)
- [Maastricht University at AMIYA: Adapting LLMs for Dialectal Arabic using Fine-tuning and MBR Decoding](https://mlsanity.com/paper/2602.09703/) (2026)
- [The Hrunting of AI: Where and How to Improve English Dialectal Fairness](https://mlsanity.com/paper/2603.15187/) (2026)
- [Can Dialects Be Steered Like Languages? Sparse Neurons and Distributed Directions in Arabic LLMs](https://mlsanity.com/paper/2607.03936/) (2026)

## Abstract (arXiv)

Dialectal Arabic (DA) varieties are under-served by language technologies, particularly large language models (LLMs). This trend threatens to exacerbate existing social inequalities and limits LLM applications, yet the research community lacks operationalized performance measurements in DA. We present a framework that comprehensively assesses LLMs' DA modeling capabilities across four dimensions: fidelity, understanding, quality, and diglossia. We evaluate nine LLMs in eight DA varieties and provide practical recommendations. Our evaluation suggests that LLMs do not produce DA as well as they understand it, not because their DA fluency is poor, but because they are reluctant to generate DA. Further analysis suggests that current post-training can contribute to bias against DA, that few-shot examples can overcome this deficiency, and that otherwise no measurable features of input text correlate well with LLM DA performance.