mlsanityData to 14 Oct 2026

EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models

arXiv 2403.10378 ↗First posted 15 Mar 2024cs.CL cs.CVcited by 41

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We introduce EXAMS-V, a new challenging multi-discipline multimodal multilingual exam benchmark for evaluating vision language models. It consists of 20,932 multiple-choice questions across 20 school disciplines covering natural science, social science, and other miscellaneous studies, e.g., religion, fine arts, business, etc. EXAMS-V includes a variety of multimodal features such as text, images, tables, figures, diagrams, maps, scientific symbols, and equations. The questions come in 11 languages from 7 language families. Unlike existing benchmarks, EXAMS-V is uniquely curated by gathering school exam questions from various countries, with a variety of education systems. This distinctive approach calls for intricate reasoning across diverse languages and relies on region-specific knowledge. Solving the problems in the dataset requires advanced perception and joint reasoning over the text and the visual content of the image. Our evaluation results demonstrate that this is a challenging dataset, which is difficult even for advanced vision-text models such as GPT-4V and Gemini; this underscores the inherent complexity of the dataset and its significance as a future benchmark.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 31 references that are in the map. First posted on arXiv 15 Mar 2024.

Builds on

20 of 31 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  11. 2019Towards VQA Models That Can Readcited by 1,718
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
8 more
  1. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  2. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  3. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  4. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 110
  5. 2023Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptationcited by 81
  6. 2019GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answeringcited by 78
  7. 2020EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answeringcited by 60
  8. 2023Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLUcited by 53

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 25; 2026: 11.

41 papers.

Sort:

By area:

  1. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 137
  2. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 134
  3. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  4. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languagescited by 55
  5. 2024VHELM: A Holistic Evaluation of Vision Language Modelscited by 53
  6. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  7. 2025VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videoscited by 24
  8. 2025Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Countingcited by 18
  9. 2024CAMEL-Bench: A Comprehensive Arabic LMM Benchmarkcited by 17
  10. 2025MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Modelscited by 15
  11. 2025MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talkscited by 14
  12. 2024M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Modelscited by 13
29 more
  1. 2024ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation?cited by 13
  2. 2025Can Vision-Language Models Evaluate Handwritten Math?cited by 10
  3. 2025IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMscited by 8
  4. 2025SemVink: Advancing VLMs' Semantic Understanding of Optical Illusions via Visual Global Thinkingcited by 6
  5. 2025CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Datasetcited by 4
  6. 2025MDK12-Bench: A Comprehensive Evaluation of Multimodal Large Language Models on Multidisciplinary Examscited by 4
  7. 2025VisTW: Benchmarking Vision-Language Models for Traditional Chinese in Taiwancited by 3
  8. 2026Do LLMs and VLMs Share Neurons for Inference? Evidence and Mechanisms of Cross-Modal Transfercited by 3
  9. 2026DISSECT: Diagnosing Where Vision Ends and Language Priors Begin in Scientific VLMscited by 3
  10. 2026LiveK12Bench: Have Large Multimodal Models Truly Conquered High School-level Examinations?cited by 3
  11. 2025Benchmarking Vision Language Models on German Factual Datacited by 2
  12. 2025VLM@school -- Evaluation of AI image understanding on German middle school knowledgecited by 2
  13. 2025UmniBench: Unified Understand and Generation Model Oriented Omni-dimensional Benchmarkcited by 2
  14. 2026M3Kang: Evaluating Multilingual Multimodal Mathematical Reasoning in Vision-Language Modelscited by 2
  15. 2026Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Modelscited by 2
  16. 2025Human-Aligned Bench: Fine-Grained Assessment of Reasoning Ability in MLLMs vs. Humanscited by 1
  17. 2025Synthetic Document Question Answering in Hungariancited by 1
  18. 2025LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluationcited by 1
  19. 2025MSA at ImageCLEF 2025 Multimodal Reasoning: Multilingual Multimodal Reasoning With Ensemble Vision Language Modelscited by 1
  20. 2025GanitBench: A bi-lingual benchmark for evaluating mathematical reasoning in Vision Language Modelscited by 1
  21. 2025Multilingual Vision-Language Models, A Surveycited by 1
  22. 2026MINERVA-Cultural: A Benchmark for Cultural and Multilingual Long Video Reasoningcited by 1
  23. 2026Test-Time Scaling for Small VLMs on Multilingual Visual MCQcited by 1
  24. 2025RoD-TAL: A Benchmark for Answering Questions in Romanian Driving License Examscited by 0
  25. 2025MELLA: Bridging Linguistic Capability and Cultural Groundedness for Low-Resource Language MLLMscited by 0
  26. 2026FAU at ImageCLEF 2026 Task on Multimodal Reasoning Robust Candidate Scoring and Concise Multilingual Visual Answeringcited by 0
  27. 2026TestHallVQA: Exploring LVLMs' Document-Level Reasoning under Redundant Contexts from Scientific Examscited by 0
  28. 2026OmniEdu: Open Foundation Models for Learning and Teachingcited by 0
  29. 2026When Can Text Replace Vision? Structural Bottlenecks in Diagram Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PISA-Bench: The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Models
  2. 2025Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation
  3. 2025Closing the Gap: Data-Centric Fine-Tuning of Vision Language Models for the Standardized Exam Questions
  4. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark
  5. 2025mmJEE-Eval: A Bilingual Multimodal Benchmark for Evaluating Scientific Reasoning in Vision-Language Models
  6. 2024Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Models
  7. 2024Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping
  8. 2025BYO-Eval: Build Your Own Dataset for Fine-Grained Visual Assessment of Multimodal Language Models