mlsanityData to 14 Oct 2026

A Survey on Benchmarks of Multimodal Large Language Models

arXiv 2408.08632 ↗First posted 16 Aug 2024cs.CL cs.AI cs.CVcited by 82

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) are gaining increasing popularity in both academia and industry due to their remarkable performance in various applications such as visual question answering, visual perception, understanding, and reasoning. Over the past few years, significant efforts have been made to examine MLLMs from multiple perspectives. This paper presents a comprehensive review of 200 benchmarks and evaluations for MLLMs, focusing on (1)perception and understanding, (2)cognition and reasoning, (3)specific domains, (4)key capabilities, and (5)other modalities. Finally, we discuss the limitations of the current evaluation methods for MLLMs and explore promising future directions. Our key argument is that evaluation should be regarded as a crucial discipline to support the development of MLLMs better. For more details, please visit our GitHub repository: https://github.com/swordlidev/Evaluation-Multimodal-LLMs-Survey.

Key facts. Cited by 82 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 100 of its 148 references that are in the map. First posted on arXiv 16 Aug 2024.

Builds on

100 of 148 references.

Closest ancestors · 23

Same direction, newest first

  1. 2024MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs24 Jun
  2. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models17 Jun
  3. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly15 Jun
  4. 2024Efficient Multimodal Large Language Models: A Survey17 May
  5. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI24 Apr
  6. 2024ImplicitAVE: An Open-Source Dataset and Multimodal LLMs Benchmark for Implicit Attribute Value Extraction24 Apr
  7. 2024Look Before You Decide: Prompting Active Deduction of MLLMs for Assumptive Reasoning19 Apr
  8. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want29 Mar
  9. 2024MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platforms21 Feb
  10. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts20 Feb
  11. 2024CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning25 Jan
  12. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences19 Jan
11 more
  1. 2024MM-SAP: A Comprehensive Benchmark for Assessing Self-Awareness of Multimodal Large Language Models in Perception15 Jan
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  3. 2023BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models5 Dec
  4. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  5. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  6. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  7. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use12 Aug
  8. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions8 Aug
  9. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  11. 2023Equivariant Similarity for Vision-Language Foundation Models25 Mar

Other references · 77

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  12. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
38 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  2. 2019Towards VQA Models That Can Readcited by 1,718
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  4. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  7. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  8. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  9. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  10. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  11. 2018Object Hallucination in Image Captioningcited by 669
  12. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  13. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  14. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  15. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
  16. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  17. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
  18. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  19. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 280
  20. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  21. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 262
  22. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  23. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 249
  24. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attackscited by 230
  25. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 226
  26. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  27. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 182
  28. 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Modelscited by 162
  29. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 135
  30. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  31. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  32. 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
  33. 2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 105
  34. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 101
  35. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  36. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 100
  37. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Modelscited by 99
  38. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 90

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 50; 2026: 19.

82 papers.

Sort:

By area:

  1. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 102
  2. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 97
  3. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Modelscited by 58
  4. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  5. 2024MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detectioncited by 47
  6. 2025MM-IFEngine: Towards Multimodal Instruction Followingcited by 42
  7. 2025MLLMs are Deeply Affected by Modality Biascited by 40
  8. 2025CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Modelscited by 35
  9. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  10. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
  11. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  12. 2025AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimizationcited by 25
38 more
  1. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  2. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveyscited by 20
  3. 2025HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluationcited by 19
  4. 2025SurveillanceVQA-589K: A Benchmark for Comprehensive Surveillance Video-Language Understanding with Large Modelscited by 18
  5. 2025VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Modelscited by 17
  6. 2025Play to Generalize: Learning to Reason Through Game Playcited by 16
  7. 2025Can Multimodal LLMs do Visual Temporal Understanding and Reasoning? The answer is No!cited by 15
  8. 2025Rethinking Driving World Model as Synthetic Data Generator for Perception Taskscited by 14
  9. 2025Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose Video Hallucination by Fine-grained Spatial-Temporal Groundingcited by 13
  10. 2025Evaluating Precise Geolocation Inference Capabilities of Vision Language Modelscited by 12
  11. 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecturecited by 11
  12. 2025Texture or Semantics? Vision-Language Models Get Lost in Font Recognitioncited by 11
  13. 2026SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenescited by 11
  14. 2024Enhanced Multimodal RAG-LLM for Accurate Visual Question Answeringcited by 10
  15. 2025A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Modelscited by 10
  16. 2025USB: A Comprehensive and Unified Safety Evaluation Benchmark for Multimodal Large Language Modelscited by 10
  17. 2025PropVG: End-to-End Proposal-Driven Visual Grounding with Multi-Granularity Discriminationcited by 10
  18. 2026DatBench: Discriminative, Faithful, and Efficient VLM Evaluationscited by 10
  19. 2024SPARK: Multi-Vision Sensor Perception and Reasoning Benchmark for Large-scale Vision-Language Modelscited by 9
  20. 2025Unified Multimodal Understanding via Byte-Pair Visual Encodingcited by 9
  21. 2026VSearcher: Long-Horizon Multimodal Search Agent via Reinforcement Learningcited by 9
  22. 2024INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuningcited by 8
  23. 20254D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object Understandingcited by 8
  24. 2025ChartGen: Scaling Chart Understanding Via Code-Guided Synthetic Chart Generationcited by 8
  25. 2025Improving Generalized Visual Grounding with Instance-aware Joint Learningcited by 8
  26. 2026Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoningcited by 8
  27. 2025UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluationcited by 7
  28. 2024Can We Predict Performance of Large Models across Vision-Language Tasks?cited by 6
  29. 2024VARCO-VISION: Expanding Frontiers in Korean Vision-Language Modelscited by 6
  30. 2025VisioMath: Benchmarking Figure-based Mathematical Reasoning in LMMscited by 6
  31. 2026Can Vision Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspectivecited by 6
  32. 2026MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcomecited by 6
  33. 2025REVAL: A Comprehension Evaluation on Reliability and Values of Large Vision-Language Modelscited by 5
  34. 2024The Labyrinth of Links: Navigating the Associative Maze of Multi-modal LLMscited by 4
  35. 2025MAPS: Multi-Agent Personality Shaping for Collaborative Reasoningcited by 4
  36. 2025Visually Interpretable Subtask Reasoning for Visual Question Answeringcited by 4
  37. 2025The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation from Recognition to Reasoningcited by 4
  38. 2025MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understandingcited by 4

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
  2. 2023MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks
  3. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  4. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria
  5. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective
  6. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
  8. 2024A Survey on Evaluation of Multimodal Large Language Models