# Beyond Cultural Knowledge: Evaluating Arabic Cultural Appropriateness of Large Language Models

arXiv: https://arxiv.org/abs/2609.16006 · HTML: https://mlsanity.com/paper/2609.16006/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 29 references that are in the map.
- First posted on arXiv 30 Jul 2026.

## Builds on (most cited first, 20 of 20)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [Probing Pre-Trained Language Models for Cross-Cultural Differences in Values](https://mlsanity.com/paper/2203.13722/) (2022)
- [The Ghost in the Machine has an American accent: value conflict in GPT-3](https://mlsanity.com/paper/2203.07785/) (2022)
- [ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic](https://mlsanity.com/paper/2402.12840/) (2024)
- [Fanar: An Arabic-Centric Multimodal Generative AI Platform](https://mlsanity.com/paper/2501.13944/) (2025)
- [How should my chatbot interact? A survey on human-chatbot interaction design](https://mlsanity.com/paper/1904.02743/) (2019)
- [ALLaM: Large Language Models for Arabic and English](https://mlsanity.com/paper/2407.15390/) (2024)
- [How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensions](https://mlsanity.com/paper/2406.14805/) (2024)
- [AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs](https://mlsanity.com/paper/2409.11404/) (2024)
- [CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeaming](https://mlsanity.com/paper/2410.02677/) (2024)
- [Jawaher: A Multidialectal Dataset of Arabic Proverbs for LLM Benchmarking](https://mlsanity.com/paper/2503.00231/) (2025)
- [Arabic Dataset for LLM Safeguard Evaluation](https://mlsanity.com/paper/2410.17040/) (2024)
- [LLMs and Cultural Values: the Impact of Prompt Language and Explicit Cultural Framing](https://mlsanity.com/paper/2511.03980/) (2025)
- [AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic](https://mlsanity.com/paper/2403.09017/) (2024)
- [Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lens](https://mlsanity.com/paper/2510.05931/) (2025)
- [CamelEval: Advancing Culturally Aligned Arabic Language Models and Benchmarks](https://mlsanity.com/paper/2409.12623/) (2024)
- [FanarGuard: A Culturally-Aware Moderation Filter for Arabic Language Models](https://mlsanity.com/paper/2511.18852/) (2025)
- [PAM: Training Policy-Aligned Moderation Filters at Scale](https://mlsanity.com/paper/2505.19766/) (2025)

## Abstract (arXiv)

Large language models (LLMs) increasingly serve users whose expectations are shaped by their cultural context, yet most cultural evaluations test what a model knows rather than how it behaves when giving open-ended recommendations, opinions, and guidance. We introduce AraBehave: 1,623 culturally grounded, open-ended Arabic prompts with 29,214 cultural-appropriateness judgments from native speakers across several Arab regions, plus a scoring model whose predictions correlate strongly with human judgments on unseen systems (Pearson r=0.74). Evaluating three Arabic-centric and three frontier LLMs, we find that cultural appropriateness is not a single capability but decomposes into two largely independent components: normative stance and grounded cultural accuracy. The best general-purpose and best Arabic-centric models score identically (3.84 vs. 3.83 of 5) yet almost never fail for the same reason: general-purpose models exhibit strong factual grounding but a culturally inappropriate normative stance, being penalized for secular framing and false balance on culturally settled matters (28--33% of their low-score rationales), while the best Arabic-centric model adopts the expected stance but is penalized for fabricated hadith and misquoted verses (29%). Stance is cheap and fragile: one sentence of cultural instruction lifts Gemini to 4.57, above every Arabic-specialized model. Conversely, a generic ``answer clearly and objectively'' prompt costs Allam-7B 0.68 points, while asking the same questions in English lowers scores for every model but one. Grounding instead tracks scale and Arabic alignment data, and disappears when culturally aware instruction tuning is replaced by a culture-neutral corpus. General safety benchmarks see none of this: they saturate above 89 while cultural scores span 2.71-3.84. We will release the benchmark, annotations, and the scoring model.