# CamelEval: Advancing Culturally Aligned Arabic Language Models and Benchmarks

arXiv: https://arxiv.org/abs/2409.12623 · HTML: https://mlsanity.com/paper/2409.12623/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 23 references that are in the map.
- First posted on arXiv 19 Sep 2024.

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [ORPO: Monolithic Preference Optimization without Reference Model](https://mlsanity.com/paper/2403.07691/) (2024)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning](https://mlsanity.com/paper/2402.06619/) (2024)
- [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](https://mlsanity.com/paper/2305.14456/) (2023)
- [Aya 23: Open Weight Releases to Further Multilingual Progress](https://mlsanity.com/paper/2405.15032/) (2024)
- [LLaMA Pro: Progressive LLaMA with Block Expansion](https://mlsanity.com/paper/2401.02415/) (2024)
- [ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic](https://mlsanity.com/paper/2402.12840/) (2024)
- [Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation](https://mlsanity.com/paper/2102.00287/) (2021)
- [Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://mlsanity.com/paper/2308.16149/) (2023)
- [CIDAR: Culturally Relevant Instruction Dataset For Arabic](https://mlsanity.com/paper/2402.03177/) (2024)

## Built on by (most cited first, 10 of 10)

- [Jawaher: A Multidialectal Dataset of Arabic Proverbs for LLM Benchmarking](https://mlsanity.com/paper/2503.00231/) (2025)
- [CARE: Multilingual Human Preference Learning for Cultural Awareness](https://mlsanity.com/paper/2504.05154/) (2025)
- [A Survey of Large Language Models for Arabic Language and its Dialects](https://mlsanity.com/paper/2410.20238/) (2024)
- [Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps](https://mlsanity.com/paper/2510.13430/) (2025)
- [MELAC: Massive Evaluation of Large Language Models with Alignment of Culture in Persian Language](https://mlsanity.com/paper/2508.00673/) (2025)
- [ALARB: An Arabic Legal Argument Reasoning Benchmark](https://mlsanity.com/paper/2510.00694/) (2025)
- [AraLingBench A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Models](https://mlsanity.com/paper/2511.14295/) (2025)
- [Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth](https://mlsanity.com/paper/2607.00139/) (2026)
- [Nuha-Speech: Building General-Purpose Arabic Speech-LLMs](https://mlsanity.com/paper/2609.11892/) (2026)
- [Beyond Cultural Knowledge: Evaluating Arabic Cultural Appropriateness of Large Language Models](https://mlsanity.com/paper/2609.16006/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are the cornerstones of modern artificial intelligence systems. This paper introduces Juhaina, a Arabic-English bilingual LLM specifically designed to align with the values and preferences of Arabic speakers. Juhaina inherently supports advanced functionalities such as instruction following, open-ended question answering, information provisioning, and text processing. Our model contains 9.24 billion parameters and is trained on a context window of up to 8,192 tokens. This paper details the creation process of Juhaina and provides an extensive empirical evaluation. Furthermore, we identify the limitations of widely-adopted Open Arabic LLM Leaderboard (OALL) and propose a new evaluation benchmark, CamelEval. Our findings demonstrate that Juhaina surpasses existing LLMs of comparable sizes, such as the Llama and Gemma families, in generating helpful responses in Arabic, providing factually accurate information about the region, and understanding nuanced cultural aspects. We aspire for Juhaina to democratize cutting-edge AI technologies, serving over 400 million Arabic speakers by offering LLMs that not only communicate in their language but also comprehend their culture. We publicly release all models on Huggingface urlhttps://huggingface.co/elmrc.