# AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic

arXiv: https://arxiv.org/abs/2403.09017 · HTML: https://mlsanity.com/paper/2403.09017/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Builds on (most cited first, 18 of 18)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://mlsanity.com/paper/2308.05374/) (2023)
- [Toxicity in ChatGPT: Analyzing Persona-assigned Language Models](https://mlsanity.com/paper/2304.05335/) (2023)
- [Multi-step Jailbreaking Privacy Attacks on ChatGPT](https://mlsanity.com/paper/2304.05197/) (2023)
- [Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment](https://mlsanity.com/paper/2308.09662/) (2023)
- [Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs](https://mlsanity.com/paper/2308.13387/) (2023)
- [CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility](https://mlsanity.com/paper/2307.09705/) (2023)
- [AceGPT, Localizing Large Language Models in Arabic](https://mlsanity.com/paper/2309.12053/) (2023)
- [ORCA: A Challenging Benchmark for Arabic Language Understanding](https://mlsanity.com/paper/2212.10758/) (2022)
- [GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP](https://mlsanity.com/paper/2305.14976/) (2023)
- [How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilities](https://mlsanity.com/paper/2311.09447/) (2023)
- [SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets](https://mlsanity.com/paper/2111.06467/) (2021)
- [SC-Safety: A Multi-round Open-ended Question Adversarial Safety Benchmark for Large Language Models in Chinese](https://mlsanity.com/paper/2310.05818/) (2023)

## Built on by (most cited first, 16 of 16)

- [Survey of Cultural Awareness in Language Models: Text and Beyond](https://mlsanity.com/paper/2411.00860/) (2024)
- [The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs](https://mlsanity.com/paper/2506.11094/) (2025)
- [Arabic Dataset for LLM Safeguard Evaluation](https://mlsanity.com/paper/2410.17040/) (2024)
- [Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps](https://mlsanity.com/paper/2510.13430/) (2025)
- [Jailbreaking LLMs with Arabic Transliteration and Arabizi](https://mlsanity.com/paper/2406.18725/) (2024)
- [Alignment at Pre-training! Towards Native Alignment for Arabic LLMs](https://mlsanity.com/paper/2412.03253/) (2024)
- [From Guidelines to Practice: A New Paradigm for Arabic Language Model Evaluation](https://mlsanity.com/paper/2506.01920/) (2025)
- [Hala Technical Report: Building Arabic-Centric Instruction & Translation Models at Scale](https://mlsanity.com/paper/2509.14008/) (2025)
- [CRiskEval: A Chinese Multi-Level Risk Evaluation Benchmark Dataset for Large Language Models](https://mlsanity.com/paper/2406.04752/) (2024)
- [AraTable: Benchmarking LLMs' Reasoning and Understanding of Arabic Tabular Data](https://mlsanity.com/paper/2507.18442/) (2025)
- [MATA: Mindful Assessment of the Telugu Abilities of Large Language Models](https://mlsanity.com/paper/2508.13526/) (2025)
- [ArabicNumBench: Evaluating Arabic Number Reading in Large Language Models](https://mlsanity.com/paper/2602.18776/) (2026)
- [Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation](https://mlsanity.com/paper/2604.03395/) (2026)
- [State-of-the-Art Arabic Language Modeling with Sparse MoE Fine-Tuning and Chain-of-Thought Distillation](https://mlsanity.com/paper/2604.06421/) (2026)
- [ArabicDialectSafety: A Dialect-Aware Benchmark for Arabic Content Safety Classification](https://mlsanity.com/paper/2608.01291/) (2026)
- [Beyond Cultural Knowledge: Evaluating Arabic Cultural Appropriateness of Large Language Models](https://mlsanity.com/paper/2609.16006/) (2026)

## Abstract (arXiv)

The swift progress and widespread acceptance of artificial intelligence (AI) systems highlight a pressing requirement to comprehend both the capabilities and potential risks associated with AI. Given the linguistic complexity, cultural richness, and underrepresented status of Arabic in AI research, there is a pressing need to focus on Large Language Models (LLMs) performance and safety for Arabic-related tasks. Despite some progress in their development, there is a lack of comprehensive trustworthiness evaluation benchmarks, which presents a major challenge in accurately assessing and improving the safety of LLMs when prompted in Arabic. In this paper, we introduce AraTrust, the first comprehensive trustworthiness benchmark for LLMs in Arabic. AraTrust comprises 522 human-written multiple-choice questions addressing diverse dimensions related to truthfulness, ethics, safety, physical health, mental health, unfairness, illegal activities, privacy, and offensive language. We evaluated a set of LLMs against our benchmark to assess their trustworthiness. GPT-4 was the most trustworthy LLM, while open-source models, particularly AceGPT 7B and Jais 13B, struggled to achieve a score of 60% in our benchmark.