mlsanityData to 13 Oct 2026

JEEM: Vision-Language Understanding in Four Arabic Dialects

arXiv 2503.21910 ↗First posted 27 Mar 2025cs.CL cs.AIcited by 7

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We introduce JEEM, a benchmark designed to evaluate Vision-Language Models (VLMs) on visual understanding across four Arabic-speaking countries: Jordan, The Emirates, Egypt, and Morocco. JEEM includes the tasks of image captioning and visual question answering, and features culturally rich and regionally diverse content. This dataset aims to assess the ability of VLMs to generalize across dialects and accurately interpret cultural elements in visual contexts. In an evaluation of five prominent open-source Arabic VLMs and GPT-4V, we find that the Arabic VLMs consistently underperform, struggling with both visual understanding and dialect-specific generation. While GPT-4V ranks best in this comparison, the model's linguistic competence varies across dialects, and its visual understanding capabilities lag behind. This underscores the need for more inclusive models and the value of culturally-diverse evaluation paradigms.

Key facts. Cited by 7 papers in the map (data through 13 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 48 references that are in the map. First posted on arXiv 27 Mar 2025.

Builds on

26 of 48 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning10 Mar
  2. 2024Maya: An Instruction Finetuned Multilingual Multimodal Model10 Dec
  3. 2024Quantifying the Gaps Between Translation and Native Perception in Training for Multimodal, Multilingual Retrieval2 Oct
  4. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark10 Jun
  5. 2023Violet: A Vision-Language Model for Arabic Image Captioning with Gemini Decoder15 Nov

Other references · 21

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  2. 2024The Llama 3 Herd of Modelscited by 14,486
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,480
  4. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,407
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,228
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,035
  7. 2021Visually Grounded Reasoning across Languages and Culturescited by 189
  8. 2019Does Object Recognition Work for Everyone?cited by 149
  9. 2021The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Modelscited by 119
  10. 2024ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabiccited by 88
  11. 2015Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answeringcited by 80
  12. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
9 more
  1. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46
  2. 2025PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Trainingcited by 38
  3. 2024PALO: A Polyglot Large Multimodal Model for 5B Peoplecited by 34
  4. 2024Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarkscited by 32
  5. 2023ALDi: Quantifying the Arabic Level of Dialectness of Textcited by 24
  6. 2024G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4ocited by 23
  7. 2024CIDAR: Culturally Relevant Instruction Dataset For Arabiccited by 17
  8. 2023Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Modelscited by 10
  9. 2018Lessons learned in multilingual grounded language learningcited by 8

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 6; 2026: 1.

7 papers.

  1. 2025Pearl: A Multimodal Culturally-Aware Arabic Instruction Datasetcited by 6
  2. 2025DialectalArabicMMLU: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Modelscited by 6
  3. 2025Hanfu-Bench: A Multimodal Benchmark on Cross-Temporal Cultural Understanding and Transcreationcited by 5
  4. 2025ARB: A Comprehensive Arabic Multimodal Reasoning Benchmarkcited by 4
  5. 2025From Words to Proverbs: Evaluating LLMs Linguistic and Cultural Competence in Saudi Dialects with Abshercited by 2
  6. 2025Low-Resource, High-Impact: Building Corpora for Inclusive Language Technologiescited by 2
  7. 2026Jako Tako or Fluent? Presenting PoVisLE: A Polish Vision-Language Evaluationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Benchmarking Vision Language Models for Cultural Understanding
  2. 2025VLURes: Benchmarking VLM Visual and Linguistic Understanding in Low-Resource Languages
  3. 2024Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Models
  4. 2025Rice-VL: Evaluating Vision-Language Models for Cultural Understanding Across ASEAN Countries
  5. 2026Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialects
  6. 2024From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models
  7. 2025BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models
  8. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understanding