# Palm: A Culturally Inclusive and Linguistically Diverse Dataset for Arabic LLMs

arXiv: https://arxiv.org/abs/2503.00151 · HTML: https://mlsanity.com/paper/2503.00151/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 28 Feb 2025.

## Builds on (most cited first, 25 of 25)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020)
- [Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning](https://mlsanity.com/paper/2402.06619/) (2024)
- [BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages](https://mlsanity.com/paper/2406.09948/) (2024)
- [Towards Measuring and Modeling "Culture" in LLMs: A Survey](https://mlsanity.com/paper/2403.15412/) (2024)
- [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](https://mlsanity.com/paper/2101.01785/) (2020)
- [No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World](https://mlsanity.com/paper/1711.08536/) (2017)
- [Investigating Cultural Alignment of Large Language Models](https://mlsanity.com/paper/2402.13231/) (2024)
- [The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models](https://mlsanity.com/paper/2103.06678/) (2021)
- [CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies](https://mlsanity.com/paper/2404.15238/) (2024)
- [A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias](https://mlsanity.com/paper/2404.00929/) (2024)
- [Extracting Cultural Commonsense Knowledge at Scale](https://mlsanity.com/paper/2210.07763/) (2022)
- [AceGPT, Localizing Large Language Models in Arabic](https://mlsanity.com/paper/2309.12053/) (2023)
- [Fanar: An Arabic-Centric Multimodal Generative AI Platform](https://mlsanity.com/paper/2501.13944/) (2025)
- [Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking](https://mlsanity.com/paper/2402.09369/) (2024)
- [ORCA: A Challenging Benchmark for Arabic Language Understanding](https://mlsanity.com/paper/2212.10758/) (2022)
- [AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs](https://mlsanity.com/paper/2409.11404/) (2024)
- [Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks](https://mlsanity.com/paper/2403.01031/) (2024)
- [CaLMQA: Exploring culturally specific long-form question answering across 23 languages](https://mlsanity.com/paper/2406.17761/) (2024)
- [NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Task](https://mlsanity.com/paper/2407.04910/) (2024)
- [Dolphin: A Challenging and Diverse Benchmark for Arabic NLG](https://mlsanity.com/paper/2305.14989/) (2023)
- [CIDAR: Culturally Relevant Instruction Dataset For Arabic](https://mlsanity.com/paper/2402.03177/) (2024)
- [JASMINE: Arabic GPT Models for Few-Shot Learning](https://mlsanity.com/paper/2212.10755/) (2022)

## Built on by (most cited first, 15 of 15)

- [CARE: Multilingual Human Preference Learning for Cultural Awareness](https://mlsanity.com/paper/2504.05154/) (2025)
- [Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps](https://mlsanity.com/paper/2510.13430/) (2025)
- [SaudiCulture: A Benchmark for Evaluating Large Language Models Cultural Competence within Saudi Arabia](https://mlsanity.com/paper/2503.17485/) (2025)
- [3LM: Bridging Arabic, STEM, and Code through Benchmarking](https://mlsanity.com/paper/2507.15850/) (2025)
- [BALSAM: A Platform for Benchmarking Arabic Large Language Models](https://mlsanity.com/paper/2507.22603/) (2025)
- [FanarGuard: A Culturally-Aware Moderation Filter for Arabic Language Models](https://mlsanity.com/paper/2511.18852/) (2025)
- [ARB: A Comprehensive Arabic Multimodal Reasoning Benchmark](https://mlsanity.com/paper/2505.17021/) (2025)
- [Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialects](https://mlsanity.com/paper/2603.21165/) (2026)
- [From Words to Proverbs: Evaluating LLMs Linguistic and Cultural Competence in Saudi Dialects with Absher](https://mlsanity.com/paper/2507.10216/) (2025)
- [Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages](https://mlsanity.com/paper/2510.05291/) (2025)
- [AdaCultureSafe: Adaptive Cultural Safety Grounded by Cultural Knowledge in Large Language Models](https://mlsanity.com/paper/2603.08275/) (2026)
- [Context-Aware Dialectal Arabic Machine Translation with Interactive Region and Register Selection](https://mlsanity.com/paper/2604.06456/) (2026)
- [Progressing beyond Art Masterpieces or Touristic Clich'es: how to assess your LLMs for cultural alignment?](https://mlsanity.com/paper/2604.25654/) (2026)
- [Enemray: Toward Capable Language Models for Hassaniya](https://mlsanity.com/paper/2609.14829/) (2026)
- [AraDynFact: Dynamic Evaluation of Factual Knowledge in Arabic](https://mlsanity.com/paper/2609.35461/) (2026)

## Abstract (arXiv)

As large language models (LLMs) become increasingly integrated into daily life, ensuring their cultural sensitivity and inclusivity is paramount. We introduce our dataset, a year-long community-driven project covering all 22 Arab countries. The dataset includes instructions (input, response pairs) in both Modern Standard Arabic (MSA) and dialectal Arabic (DA), spanning 20 diverse topics. Built by a team of 44 researchers across the Arab world, all of whom are authors of this paper, our dataset offers a broad, inclusive perspective. We use our dataset to evaluate the cultural and dialectal capabilities of several frontier LLMs, revealing notable limitations. For instance, while closed-source LLMs generally exhibit strong performance, they are not without flaws, and smaller open-source models face greater challenges. Moreover, certain countries (e.g., Egypt, the UAE) appear better represented than others (e.g., Iraq, Mauritania, Yemen). Our annotation guidelines, code, and data for reproducibility are publicly available.