mlsanityData to 14 Oct 2026

3LM: Bridging Arabic, STEM, and Code through Benchmarking

arXiv 2507.15850 ↗First posted 21 Jul 2025cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Arabic is one of the most widely spoken languages in the world, yet efforts to develop and evaluate Large Language Models (LLMs) for Arabic remain relatively limited. Most existing Arabic benchmarks focus on linguistic, cultural, or religious content, leaving a significant gap in domains like STEM and code which are increasingly relevant for real-world LLM applications. To help bridge this gap, we present 3LM, a suite of three benchmarks designed specifically for Arabic. The first is a set of STEM-related question-answer pairs, naturally sourced from Arabic textbooks and educational worksheets. The second consists of synthetically generated STEM questions, created using the same sources. The third benchmark focuses on code generation, built through a careful translation of two widely used code benchmarks, incorporating a human-in-the-loop process with several rounds of review to ensure high-quality and faithful translations. We release all three benchmarks publicly to support the growth of Arabic LLM research in these essential but underrepresented areas.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 15 of its 26 references that are in the map. First posted on arXiv 21 Jul 2025.

Builds on

15 of 26 references.

References · 15

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  5. 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 86
  6. 2023AceGPT, Localizing Large Language Models in Arabiccited by 75
  7. 2022ORCA: A Challenging Benchmark for Arabic Language Understandingcited by 47
  8. 2024AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMscited by 46
  9. 2024mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generationcited by 32
  10. 2025YourBench: Easy Custom Evaluation Sets for Everyonecited by 20
  11. 2025Jawaher: A Multidialectal Dataset of Arabic Proverbs for LLM Benchmarkingcited by 18
  12. 2025Fann or Flop: A Multigenre, Multiera Benchmark for Arabic Poetry Understanding in LLMscited by 16
3 more
  1. 2025Palm: A Culturally Inclusive and Linguistically Diverse Dataset for Arabic LLMscited by 15
  2. 2024AraSTEM: A Native Arabic Multiple Choice Question Benchmark for Evaluating LLMs Knowledge In STEM Subjectscited by 7
  3. 2025From Guidelines to Practice: A New Paradigm for Arabic Language Model Evaluationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gapscited by 9
  2. 2025DialectalArabicMMLU: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Modelscited by 6
  3. 2025AraLingBench A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Modelscited by 2
  4. 2026Bridging the English-Arabic Medical Knowledge Gap: Targeted Low-Rank Adaptation via Causal Layer Selectioncited by 0
  5. 2026AraDynFact: Dynamic Evaluation of Factual Knowledge in Arabiccited by 0