# Arabic Dataset for LLM Safeguard Evaluation

arXiv: https://arxiv.org/abs/2410.17040 · HTML: https://mlsanity.com/paper/2410.17040/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 22 Oct 2024.

## Builds on (most cited first, 13 of 13)

- [Low-Resource Languages Jailbreak GPT-4](https://mlsanity.com/paper/2310.02446/) (2023)
- [Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment](https://mlsanity.com/paper/2308.09662/) (2023)
- [The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts](https://mlsanity.com/paper/2401.13136/) (2024)
- [AceGPT, Localizing Large Language Models in Arabic](https://mlsanity.com/paper/2309.12053/) (2023)
- [Against The Achilles' Heel: A Survey on Red Teaming for Generative Models](https://mlsanity.com/paper/2404.00629/) (2024)
- [ORCA: A Challenging Benchmark for Arabic Language Understanding](https://mlsanity.com/paper/2212.10758/) (2022)
- [RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios?](https://mlsanity.com/paper/2404.14397/) (2024)
- [Unveiling Safety Vulnerabilities of Large Language Models](https://mlsanity.com/paper/2311.04124/) (2023)
- [AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic](https://mlsanity.com/paper/2403.09017/) (2024)
- [Emojis as Anchors to Detect Arabic Offensive Language and Hate Speech](https://mlsanity.com/paper/2201.06723/) (2022)
- [Understanding and Detecting Dangerous Speech in Social Media](https://mlsanity.com/paper/2005.06608/) (2020)
- [Analyzing Multilingual Competency of LLMs in Multi-Turn Instruction Following: A Case Study of Arabic](https://mlsanity.com/paper/2310.14819/) (2023)
- [Toxic language detection: a systematic review of Arabic datasets](https://mlsanity.com/paper/2312.07228/) (2023)

## Built on by (most cited first, 17 of 17)

- [Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps](https://mlsanity.com/paper/2510.13430/) (2025)
- [IndoSafety: Culturally Grounded Safety for LLMs in Indonesian Languages](https://mlsanity.com/paper/2506.02573/) (2025)
- [BALSAM: A Platform for Benchmarking Arabic Large Language Models](https://mlsanity.com/paper/2507.22603/) (2025)
- [Qorgau: Evaluating LLM Safety in Kazakh-Russian Bilingual Contexts](https://mlsanity.com/paper/2502.13640/) (2025)
- [PAM: Training Policy-Aligned Moderation Filters at Scale](https://mlsanity.com/paper/2505.19766/) (2025)
- [Toxicity Red-Teaming: Benchmarking LLM Safety in Singapore's Low-Resource Languages](https://mlsanity.com/paper/2509.15260/) (2025)
- [CRaFT: An Explanation-Based Framework for Evaluating Cultural Reasoning in Multilingual Language Models](https://mlsanity.com/paper/2510.14014/) (2025)
- [On The Conceptualization and Societal Impact of Cross-Cultural Bias](https://mlsanity.com/paper/2512.21809/) (2025)
- [AdaCultureSafe: Adaptive Cultural Safety Grounded by Cultural Knowledge in Large Language Models](https://mlsanity.com/paper/2603.08275/) (2026)
- [To Lie or Not to Lie? Investigating The Biased Spread of Global Lies by LLMs](https://mlsanity.com/paper/2604.06552/) (2026)
- [Sch"utzen: Evaluating LLM Safety in Bulgarian and German Contexts](https://mlsanity.com/paper/2606.11316/) (2026)
- [ArabicDialectSafety: A Dialect-Aware Benchmark for Arabic Content Safety Classification](https://mlsanity.com/paper/2608.01291/) (2026)
- [Redteaming Leading Arabic LLMs with ASAS](https://mlsanity.com/paper/2608.21985/) (2026)
- [Arabic Safety Alignment as Selective Refusal: An Empirical Study of SFT, DPO, and Guard Calibration](https://mlsanity.com/paper/2608.29378/) (2026)
- [Beyond Cultural Knowledge: Evaluating Arabic Cultural Appropriateness of Large Language Models](https://mlsanity.com/paper/2609.16006/) (2026)
- [CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systems](https://mlsanity.com/paper/2609.22494/) (2026)
- [ArGuard Shared Task: Harmful Content Detection in Arabic Memes and LLM Prompts](https://mlsanity.com/paper/2609.29349/) (2026)

## Abstract (arXiv)

The growing use of large language models (LLMs) has raised concerns regarding their safety. While many studies have focused on English, the safety of LLMs in Arabic, with its linguistic and cultural complexities, remains under-explored. Here, we aim to bridge this gap. In particular, we present an Arab-region-specific safety evaluation dataset consisting of 5,799 questions, including direct attacks, indirect attacks, and harmless requests with sensitive words, adapted to reflect the socio-cultural context of the Arab world. To uncover the impact of different stances in handling sensitive and controversial topics, we propose a dual-perspective evaluation framework. It assesses the LLM responses from both governmental and opposition viewpoints. Experiments over five leading Arabic-centric and multilingual LLMs reveal substantial disparities in their safety performance. This reinforces the need for culturally specific datasets to ensure the responsible deployment of LLMs.