# AraReasoner: Evaluating Reasoning-Based LLMs for Arabic NLP

arXiv: https://arxiv.org/abs/2506.08768 · HTML: https://mlsanity.com/paper/2506.08768/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 15 of its 34 references that are in the map.
- First posted on arXiv 10 Jun 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 15 of 15)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [LoRA+: Efficient Low Rank Adaptation of Large Models](https://mlsanity.com/paper/2402.12354/) (2024)
- [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020)
- [AraT5: Text-to-Text Transformers for Arabic Language Generation](https://mlsanity.com/paper/2109.12068/) (2021)
- [Empowering LLMs with Logical Reasoning: A Comprehensive Survey](https://mlsanity.com/paper/2502.15652/) (2025)
- [ORCA: A Challenging Benchmark for Arabic Language Understanding](https://mlsanity.com/paper/2212.10758/) (2022)
- [Stance Prediction and Claim Verification: An Arabic Perspective](https://mlsanity.com/paper/2005.10410/) (2020)
- [LAraBench: Benchmarking Arabic AI with Large Language Models](https://mlsanity.com/paper/2305.14982/) (2023)
- [Deep Diacritization: Efficient Hierarchical Recurrence for Improved Arabic Diacritization](https://mlsanity.com/paper/2011.00538/) (2020)
- [Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models](https://mlsanity.com/paper/2306.16322/) (2023)

## Built on by (most cited first, 5 of 5)

- [AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMs](https://mlsanity.com/paper/2509.04656/) (2025)
- [From Words to Proverbs: Evaluating LLMs Linguistic and Cultural Competence in Saudi Dialects with Absher](https://mlsanity.com/paper/2507.10216/) (2025)
- [AraTable: Benchmarking LLMs' Reasoning and Understanding of Arabic Tabular Data](https://mlsanity.com/paper/2507.18442/) (2025)
- [ViPo-MLLM: Visual-Pose Multimodal LLM for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2607.03657/) (2026)
- [Halluscoring 2026: The first shared task on llms hallucination detection and answer verification](https://mlsanity.com/paper/2609.38355/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown remarkable progress in reasoning abilities and general natural language processing (NLP) tasks, yet their performance on Arabic data, characterized by rich morphology, diverse dialects, and complex script, remains underexplored. This paper presents a comprehensive benchmarking study of multiple reasoning-focused LLMs, with a special emphasis on the newly introduced DeepSeek models, across a suite of fifteen Arabic NLP tasks. We experiment with various strategies, including zero-shot, few-shot, and fine-tuning. This allows us to systematically evaluate performance on datasets covering a range of applications to examine their capacity for linguistic reasoning under different levels of complexity. Our experiments reveal several key findings. First, carefully selecting just three in-context examples delivers an average uplift of over 13 F1 points on classification tasks-boosting sentiment analysis from 35.3% to 87.5% and paraphrase detection from 56.1% to 87.0%. Second, reasoning-focused DeepSeek architectures outperform a strong GPT o4-mini baseline by an average of 12 F1 points on complex inference tasks in the zero-shot setting. Third, LoRA-based fine-tuning yields up to an additional 8 points in F1 and BLEU compared to equivalent increases in model scale. The code is available at https://github.com/gufranSabri/deepseek-evals