mlsanityData to 14 Oct 2026

M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models

arXiv 2306.05179 ↗First posted 8 Jun 2023cs.CL cs.CVcited by 110

Not on the map. Why?

Cited by 37 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite the existence of various benchmarks for evaluating natural language processing models, we argue that human exams are a more suitable means of evaluating general intelligence for large language models (LLMs), as they inherently demand a much wider range of abilities such as language understanding, domain knowledge, and problem-solving skills. To this end, we introduce M3Exam, a novel benchmark sourced from real and official human exam questions for evaluating LLMs in a multilingual, multimodal, and multilevel context. M3Exam exhibits three unique characteristics: (1) multilingualism, encompassing questions from multiple countries that require strong multilingual proficiency and cultural knowledge; (2) multimodality, accounting for the multimodal nature of many exam questions to test the model's multimodal understanding capability; and (3) multilevel structure, featuring exams from three critical educational periods to comprehensively assess a model's proficiency at different levels. In total, M3Exam contains 12,317 questions in 9 diverse languages with three educational levels, where about 23% of the questions require processing images for successful solving. We assess the performance of top-performing LLMs on M3Exam and find that current models, including GPT-4, still struggle with multilingual text, particularly in low-resource and non-Latin script languages. Multimodal LLMs also perform poorly with complex multimodal questions. We believe that M3Exam can be a valuable resource for comprehensively evaluating LLMs by examining their multilingual and multimodal abilities and tracking their development. Data and evaluation code is available at urlhttps://github.com/DAMO-NLP-SG/M3Exam.

Key facts. Cited by 110 papers in the map (data through 14 Oct 2026). Builds on 34 of its 45 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

34 of 45 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
22 more
  1. 2015VQA: Visual Question Answeringcited by 2,506
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  6. 2023PaLM 2 Technical Reportcited by 1,162
  7. 2023BloombergGPT: A Large Language Model for Financecited by 939
  8. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  9. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  10. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  11. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  12. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  13. 2018nocaps: novel object captioning at scalecited by 444
  14. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  15. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  16. 2022Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofscited by 291
  17. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  18. 2021Visually Grounded Reasoning across Languages and Culturescited by 189
  19. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 186
  20. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  21. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  22. 2023Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinationscited by 63

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 17; 2024: 38; 2025: 41; 2026: 14.

110 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  2. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
  3. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  4. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 240
  5. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  6. 2024Large Language Models for Education: A Survey and Outlookcited by 188
  7. 2024How do Large Language Models Handle Multilingualism?cited by 150
  8. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 146
  9. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  10. 2023CMB: A Comprehensive Medical Benchmark in Chinesecited by 115
  11. 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
  12. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 97
38 more
  1. 2023SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoningcited by 94
  2. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  3. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 64
  4. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 64
  5. 2024CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligencecited by 62
  6. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languagescited by 55
  7. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  8. 2024EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Modelscited by 41
  9. 2023CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generationcited by 39
  10. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 35
  11. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 35
  12. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  13. 2024CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoningcited by 31
  14. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  15. 2023Typhoon: Thai Large Language Modelscited by 30
  16. 2024Parrot: Multilingual Visual Instruction Tuningcited by 30
  17. 2024SPRIG: Improving Large Language Model Performance by System Prompt Optimizationcited by 29
  18. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  19. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  20. 2025MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languagescited by 26
  21. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  22. 2023ChatGPT MT: Competitive for High- (but not Low-) Resource Languagescited by 23
  23. 2024Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?cited by 22
  24. 2025Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulationscited by 21
  25. 2024Sailor: Open Language Models for South-East Asiacited by 19
  26. 2025HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluationcited by 19
  27. 2023BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Modelscited by 18
  28. 2024Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languagescited by 18
  29. 2024Eureka: Evaluating and Understanding Large Foundation Modelscited by 18
  30. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 18
  31. 2024II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Modelscited by 16
  32. 2025Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Trainingcited by 16
  33. 2024M3GIA: A Cognition Inspired Multilingual and Multimodal General Intelligence Ability Benchmarkcited by 14
  34. 2024Can MLLMs Understand the Deep Implication Behind Chinese Images?cited by 14
  35. 2025MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talkscited by 14
  36. 2024DevBench: A multimodal developmental benchmark for language learningcited by 13
  37. 2024Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Modelscited by 12
  38. 2025MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understandingcited by 12

50 most cited of 110 All 110 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MDK12-Bench: A Comprehensive Evaluation of Multimodal Large Language Models on Multidisciplinary Exams
  2. 2025MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
  4. 2025MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models
  5. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation
  6. 2024M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models
  7. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
  8. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Models