# Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Tasks

arXiv: https://arxiv.org/abs/2602.05374 · HTML: https://mlsanity.com/paper/2602.05374/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 5 Feb 2026.

## Builds on (most cited first, 18 of 18)

- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Benchmarking Retrieval-Augmented Generation for Medicine](https://mlsanity.com/paper/2402.13178/) (2024)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://mlsanity.com/paper/2402.10373/) (2024)
- [HuatuoGPT, towards Taming Language Model to Be a Doctor](https://mlsanity.com/paper/2305.15075/) (2023)
- [The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models](https://mlsanity.com/paper/2103.06678/) (2021)
- [MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphs](https://mlsanity.com/paper/2504.00993/) (2025)
- [Towards Building Multilingual Language Model for Medicine](https://mlsanity.com/paper/2402.13963/) (2024)
- [Med42-v2: A Suite of Clinical LLMs](https://mlsanity.com/paper/2408.06142/) (2024)
- [Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries](https://mlsanity.com/paper/2310.13132/) (2023)
- [MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering](https://mlsanity.com/paper/2404.05590/) (2024)
- [Medical Exam Question Answering with Large-scale Reading Comprehension](https://mlsanity.com/paper/1802.10279/) (2018)
- [Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?](https://mlsanity.com/paper/2411.04118/) (2024)
- [MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks](https://mlsanity.com/paper/2505.03427/) (2025)
- [Facts Fade Fast: Evaluating Memorization of Outdated Medical Knowledge in Large Language Models](https://mlsanity.com/paper/2509.04304/) (2025)
- [Are LLMs Truly Multilingual? Exploring Zero-Shot Multilingual Capability of LLMs for Information Retrieval: An Italian Healthcare Use Case](https://mlsanity.com/paper/2512.04834/) (2025)
- [When Are Names Similar Or the Same? Introducing the Code Names Matcher Library](https://mlsanity.com/paper/2209.03198/) (2022)

## Built on by (most cited first, 2 of 2)

- [Biomedical Machine Translation for Low-Resource Arabic-Script Languages via Cross-Lingual Transfer and LoRA Adapter Merging](https://mlsanity.com/paper/2607.22300/) (2026)
- [Bridging the English-Arabic Medical Knowledge Gap: Targeted Low-Rank Adaptation via Causal Layer Selection](https://mlsanity.com/paper/2608.00207/) (2026)

## Abstract (arXiv)

In recent years, Large Language Models (LLMs) have become widely used in medical applications, such as clinical decision support, medical education, and medical question answering. Yet, these models are often English-centric, limiting their robustness and reliability for linguistically diverse communities. Recent work has highlighted discrepancies in performance in low-resource languages for various medical tasks, but the underlying causes remain poorly understood. In this study, we conduct a cross-lingual empirical analysis of LLM performance on Arabic and English medical question and answering. Our findings reveal a persistent language-driven performance gap that intensifies with increasing task complexity. Tokenization analysis exposes structural fragmentation in Arabic medical text, while reliability analysis suggests that model-reported confidence and explanations exhibit limited correlation with correctness. Together, these findings underscore the need for language-aware design and evaluation strategies in LLMs for medical tasks.