# EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models

arXiv: https://arxiv.org/abs/2403.10378 · HTML: https://mlsanity.com/paper/2403.10378/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning](https://mlsanity.com/paper/2102.10407/) (2021)
- [M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models](https://mlsanity.com/paper/2306.05179/) (2023)
- [Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation](https://mlsanity.com/paper/2305.15011/) (2023)
- [GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering](https://mlsanity.com/paper/1902.09506/) (2019)
- [EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answering](https://mlsanity.com/paper/2011.03080/) (2020)
- [Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU](https://mlsanity.com/paper/2310.04928/) (2023)

## Built on by (most cited first, 30 of 41)

- [MMVU: Measuring Expert-Level Multi-Discipline Video Understanding](https://mlsanity.com/paper/2501.12380/) (2025)
- [Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark](https://mlsanity.com/paper/2501.05444/) (2025)
- [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025)
- [All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages](https://mlsanity.com/paper/2411.16508/) (2024)
- [VHELM: A Holistic Evaluation of Vision Language Models](https://mlsanity.com/paper/2410.07112/) (2024)
- [Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning](https://mlsanity.com/paper/2502.02871/) (2025)
- [VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos](https://mlsanity.com/paper/2506.10857/) (2025)
- [Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Counting](https://mlsanity.com/paper/2510.04401/) (2025)
- [CAMEL-Bench: A Comprehensive Arabic LMM Benchmark](https://mlsanity.com/paper/2410.18976/) (2024)
- [MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2504.05782/) (2025)
- [MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks](https://mlsanity.com/paper/2507.19634/) (2025)
- [M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models](https://mlsanity.com/paper/2405.15638/) (2024)
- [ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation?](https://mlsanity.com/paper/2412.02368/) (2024)
- [Can Vision-Language Models Evaluate Handwritten Math?](https://mlsanity.com/paper/2501.07244/) (2025)
- [IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs](https://mlsanity.com/paper/2511.04727/) (2025)
- [SemVink: Advancing VLMs' Semantic Understanding of Optical Illusions via Visual Global Thinking](https://mlsanity.com/paper/2506.02803/) (2025)
- [CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Dataset](https://mlsanity.com/paper/2507.22752/) (2025)
- [MDK12-Bench: A Comprehensive Evaluation of Multimodal Large Language Models on Multidisciplinary Exams](https://mlsanity.com/paper/2508.06851/) (2025)
- [VisTW: Benchmarking Vision-Language Models for Traditional Chinese in Taiwan](https://mlsanity.com/paper/2503.10427/) (2025)
- [Do LLMs and VLMs Share Neurons for Inference? Evidence and Mechanisms of Cross-Modal Transfer](https://mlsanity.com/paper/2602.19058/) (2026)
- [DISSECT: Diagnosing Where Vision Ends and Language Priors Begin in Scientific VLMs](https://mlsanity.com/paper/2604.06250/) (2026)
- [LiveK12Bench: Have Large Multimodal Models Truly Conquered High School-level Examinations?](https://mlsanity.com/paper/2605.26781/) (2026)
- [Benchmarking Vision Language Models on German Factual Data](https://mlsanity.com/paper/2504.11108/) (2025)
- [VLM@school -- Evaluation of AI image understanding on German middle school knowledge](https://mlsanity.com/paper/2506.11604/) (2025)
- [UmniBench: Unified Understand and Generation Model Oriented Omni-dimensional Benchmark](https://mlsanity.com/paper/2512.17196/) (2025)
- [M3Kang: Evaluating Multilingual Multimodal Mathematical Reasoning in Vision-Language Models](https://mlsanity.com/paper/2601.16218/) (2026)
- [Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Models](https://mlsanity.com/paper/2601.22737/) (2026)
- [Human-Aligned Bench: Fine-Grained Assessment of Reasoning Ability in MLLMs vs. Humans](https://mlsanity.com/paper/2505.11141/) (2025)
- [Synthetic Document Question Answering in Hungarian](https://mlsanity.com/paper/2505.23008/) (2025)
- [LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluation](https://mlsanity.com/paper/2507.07274/) (2025)

## Abstract (arXiv)

We introduce EXAMS-V, a new challenging multi-discipline multimodal multilingual exam benchmark for evaluating vision language models. It consists of 20,932 multiple-choice questions across 20 school disciplines covering natural science, social science, and other miscellaneous studies, e.g., religion, fine arts, business, etc. EXAMS-V includes a variety of multimodal features such as text, images, tables, figures, diagrams, maps, scientific symbols, and equations. The questions come in 11 languages from 7 language families. Unlike existing benchmarks, EXAMS-V is uniquely curated by gathering school exam questions from various countries, with a variety of education systems. This distinctive approach calls for intricate reasoning across diverse languages and relies on region-specific knowledge. Solving the problems in the dataset requires advanced perception and joint reasoning over the text and the visual content of the image. Our evaluation results demonstrate that this is a challenging dataset, which is difficult even for advanced vision-text models such as GPT-4V and Gemini; this underscores the inherent complexity of the dataset and its significance as a future benchmark.