# A Comprehensive Evaluation of Large Language Models on Mental Illnesses in Arabic Context

arXiv: https://arxiv.org/abs/2501.06859 · HTML: https://mlsanity.com/paper/2501.06859/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 35 references that are in the map.
- First posted on arXiv 12 Jan 2025.

## Builds on (most cited first, 8 of 8)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Dreaddit: A Reddit Dataset for Stress Analysis in Social Media](https://mlsanity.com/paper/1911.00133/) (2019)
- [Deep Learning for Suicide and Depression Identification with Unsupervised Label Correction](https://mlsanity.com/paper/2102.09427/) (2021)
- [Data set creation and empirical analysis for detecting signs of depression from social media postings](https://mlsanity.com/paper/2202.03047/) (2022)
- [BiMediX: Bilingual Medical Mixture of Experts LLM](https://mlsanity.com/paper/2402.13253/) (2024)
- [DEPTWEET: A Typology for Social Media Texts to Detect Depression Severities](https://mlsanity.com/paper/2210.05372/) (2022)
- [A Comprehensive Evaluation of Large Language Models on Mental Illnesses](https://mlsanity.com/paper/2409.15687/) (2024)

## Built on by (most cited first, 3 of 3)

- [A Survey on Multilingual Mental Disorders Detection from Social Media Data](https://mlsanity.com/paper/2505.15556/) (2025)
- [A Comprehensive Review of Datasets for Clinical Mental Health AI Systems](https://mlsanity.com/paper/2508.09809/) (2025)
- [Large Language Models for Mental Health: A Multilingual Evaluation](https://mlsanity.com/paper/2602.02440/) (2026)

## Abstract (arXiv)

Mental health disorders pose a growing public health concern in the Arab world, emphasizing the need for accessible diagnostic and intervention tools. Large language models (LLMs) offer a promising approach, but their application in Arabic contexts faces challenges including limited labeled datasets, linguistic complexity, and translation biases. This study comprehensively evaluates 8 LLMs, including general multi-lingual models, as well as bi-lingual ones, on diverse mental health datasets (such as AraDepSu, Dreaddit, MedMCQA), investigating the impact of prompt design, language configuration (native Arabic vs. translated English, and vice versa), and few-shot prompting on diagnostic performance. We find that prompt engineering significantly influences LLM scores mainly due to reduced instruction following, with our structured prompt outperforming a less structured variant on multi-class datasets, with an average difference of 14.5%. While language influence on performance was modest, model selection proved crucial: Phi-3.5 MoE excelled in balanced accuracy, particularly for binary classification, while Mistral NeMo showed superior performance in mean absolute error for severity prediction tasks. Few-shot prompting consistently improved performance, with particularly substantial gains observed for GPT-4o Mini on multi-class classification, boosting accuracy by an average factor of 1.58. These findings underscore the importance of prompt optimization, multilingual analysis, and few-shot learning for developing culturally sensitive and effective LLM-based mental health tools for Arabic-speaking populations.