mlsanityData to 14 Oct 2026

ImageEval 2026: Culturally Grounded Arabic Multimodal Evaluation

arXiv 2608.30475 ↗First posted 31 Aug 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We present an overview of the ImageEval 2026 shared task on culturally grounded Arabic multimodal evaluation. It includes two tasks: (i) AynVQA, covering spoken visual question answering and image-grounded hallucination detection in English and Modern Standard Arabic (MSA), and (ii) CRAI-Bench, evaluating the cultural accuracy of text-to-image generation. A total of 14 teams participated in the test phase, with 12 teams submitting system description papers. Participating systems used a range of approaches, including zero-shot prompting, fine-tuning of vision-language models, speech-recognition pipelines, ensembling, and score calibration. We describe the task setup, datasets, evaluation procedure, and participating systems, and summarize the main results across the different tracks. All datasets and evaluation scripts from the shared task are released to the research community. The shared task highlights the challenges of culturally grounded multimodal evaluation, particularly for Arabic speech and image-text reasoning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 53 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

27 of 53 references.

References · 27

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  3. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 300
  4. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmarkcited by 120
  5. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 103
  6. 2024Benchmarking Vision Language Models for Cultural Understandingcited by 101
  7. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  8. 2018Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehensioncited by 74
  9. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  10. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewritescited by 61
  11. 2022End-to-end Spoken Conversational Question Answering: Task, Dataset and Modelcited by 37
  12. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 27
15 more
  1. 2023Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Modelscited by 23
  2. 2024CAMEL-Bench: A Comprehensive Arabic LMM Benchmarkcited by 17
  3. 2024Dallah: A Dialect-Aware Multimodal Large Language Model for Arabiccited by 16
  4. 2025CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metricscited by 15
  5. 2025Diffusion Models Through a Global Lens: Are They Culturally Inclusive?cited by 14
  6. 2025The Landscape of Arabic Large Language Models (ALLMs): A New Era for Arabic Language Technologycited by 12
  7. 2025AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMscited by 12
  8. 2025Culture in Action: Evaluating Text-to-Image Models through Social Activitiescited by 10
  9. 2025SpokenNativQA: Multilingual Everyday Spoken Queries for LLMscited by 8
  10. 2025When Cultures Meet: Multicultural Text-to-Image Generationcited by 7
  11. 2025Pearl: A Multimodal Culturally-Aware Arabic Instruction Datasetcited by 6
  12. 2025OASIS: A Multilingual and Multimodal Dataset for Culturally Grounded Spoken Visual QAcited by 5
  13. 2026Evaluating AI-Generated Images of Cultural Artifacts with Community-Informed Rubricscited by 3
  14. 2026Said Aloud, Read Different: Cross-Modal Instability in Multimodal Modelscited by 2
  15. 2026WASIL: In-the-Wild Arabic Spoken Interactions with LLMscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025ARB: A Comprehensive Arabic Multimodal Reasoning Benchmark
  2. 2025JEEM: Vision-Language Understanding in Four Arabic Dialects
  3. 2025IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs
  4. 2025Grounding Multilingual Multimodal LLMs With Cultural Knowledge
  5. 2026Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialects
  6. 2026Once Correct, Still Wrong: Counterfactual Hallucination in Multilingual Vision-Language Models
  7. 2026UEval: A Benchmark for Unified Multimodal Generation
  8. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages