# AraSTEM: A Native Arabic Multiple Choice Question Benchmark for Evaluating LLMs Knowledge In STEM Subjects

arXiv: https://arxiv.org/abs/2501.00559 · HTML: https://mlsanity.com/paper/2501.00559/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2024.

## Built on by (most cited first, 7 of 7)

- [MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks](https://mlsanity.com/paper/2505.03427/) (2025)
- [Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps](https://mlsanity.com/paper/2510.13430/) (2025)
- [3LM: Bridging Arabic, STEM, and Code through Benchmarking](https://mlsanity.com/paper/2507.15850/) (2025)
- [ARB: A Comprehensive Arabic Multimodal Reasoning Benchmark](https://mlsanity.com/paper/2505.17021/) (2025)
- [AraHealthQA 2025: The First Shared Task on Arabic Health Question Answering](https://mlsanity.com/paper/2508.20047/) (2025)
- [ALPS: A Diagnostic Challenge Set for Arabic Linguistic & Pragmatic Reasoning](https://mlsanity.com/paper/2602.17054/) (2026)
- [Benchmarking the Medical Understanding and Reasoning of Large Language Models in Arabic Healthcare Tasks](https://mlsanity.com/paper/2508.15797/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) have shown remarkable capabilities, not only in generating human-like text, but also in acquiring knowledge. This highlights the need to go beyond the typical Natural Language Processing downstream benchmarks and asses the various aspects of LLMs including knowledge and reasoning. Numerous benchmarks have been developed to evaluate LLMs knowledge, but they predominantly focus on the English language. Given that many LLMs are multilingual, relying solely on benchmarking English knowledge is insufficient. To address this issue, we introduce AraSTEM, a new Arabic multiple-choice question dataset aimed at evaluating LLMs knowledge in STEM subjects. The dataset spans a range of topics at different levels which requires models to demonstrate a deep understanding of scientific Arabic in order to achieve high accuracy. Our findings show that publicly available models of varying sizes struggle with this dataset, and underscores the need for more localized language models. The dataset is freely accessible on Hugging Face.