mlsanityData to 7 Oct 2026

A Survey on Multimodal Benchmarks: In the Era of Large AI Models

arXiv 2409.18142 ↗First posted 21 Sep 2024cs.AI cs.MMcited by 27

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The rapid evolution of Multimodal Large Language Models (MLLMs) has brought substantial advancements in artificial intelligence, significantly enhancing the capability to understand and generate multimodal content. While prior studies have largely concentrated on model architectures and training methodologies, a thorough analysis of the benchmarks used for evaluating these models remains underexplored. This survey addresses this gap by systematically reviewing 211 benchmarks that assess MLLMs across four core domains: understanding, reasoning, generation, and application. We provide a detailed analysis of task designs, evaluation metrics, and dataset constructions, across diverse modalities. We hope that this survey will contribute to the ongoing advancement of MLLM research by offering a comprehensive overview of benchmarking practices and identifying promising directions for future work. An associated GitHub repository collecting the latest papers is available.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 223 of its 246 references that are in the map. First posted on arXiv 21 Sep 2024.

Builds on

223 of 246 references.

Closest ancestors · 55

Same direction, newest first

  1. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models5 Aug
  2. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception11 Jul
  3. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts6 Jul
  4. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs1 Jul
  5. 2024MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs24 Jun
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  7. 2024LLaNA: Large Language and NeRF Assistant17 Jun
  8. 2024MFC-Bench: Benchmarking Multimodal Fact-Checking with Large Vision-Language Models17 Jun
  9. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models17 Jun
  10. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly15 Jun
  11. 2024What is the Visual Cognition Gap between Humans and Multimodal LLMs?14 Jun
  12. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding13 Jun
38 more
  1. 2024Needle In A Multimodal Haystack11 Jun
  2. 2024II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models9 Jun
  3. 2024Wings: Learning Multimodal LLMs without Text-only Forgetting5 Jun
  4. 2024Parrot: Multilingual Visual Instruction Tuning4 Jun
  5. 2024MANTIS: Interleaved Multi-Image Instruction Tuning2 May
  6. 2024MileBench: Benchmarking MLLMs in Long Context29 Apr
  7. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension25 Apr
  8. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI24 Apr
  9. 2024ImplicitAVE: An Open-Source Dataset and Multimodal LLMs Benchmark for Implicit Attribute Value Extraction24 Apr
  10. 2024Look Before You Decide: Prompting Active Deduction of MLLMs for Assumptive Reasoning19 Apr
  11. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  12. 2024LaVy: Vietnamese Multimodal Large Language Model11 Apr
  13. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want29 Mar
  14. 2024Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks1 Mar
  15. 2024GenCeption: Evaluate Vision LLMs with Unlabeled Unimodal Data22 Feb
  16. 2024MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platforms21 Feb
  17. 2024CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models21 Feb
  18. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts20 Feb
  19. 2024Model Composition for Multimodal Large Language Models20 Feb
  20. 2024MULTI: Multimodal Understanding Leaderboard with Text and Images5 Feb
  21. 2024CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning25 Jan
  22. 2024CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark22 Jan
  23. 2024Benchmarking Large Multimodal Models against Common Corruptions22 Jan
  24. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences19 Jan
  25. 2024MM-SAP: A Comprehensive Benchmark for Assessing Self-Awareness of Multimodal Large Language Models in Perception15 Jan
  26. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  27. 2024TransportationGames: Benchmarking Transportation Knowledge of (Multimodal) Large Language Models9 Jan
  28. 2023BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models5 Dec
  29. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models28 Nov
  30. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models5 Nov
  31. 2023Can We Edit Multimodal Large Language Models?12 Oct
  32. 2023TouchStone: Evaluating Vision-Language Models by Language Models31 Aug
  33. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use12 Aug
  34. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions8 Aug
  35. 2023TinyLVLM-eHub: Towards Comprehensive and Efficient Evaluation for Large Vision-Language Models7 Aug
  36. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  37. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  38. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

50 newest of 55 All 55 (JSON)

Other references · 168

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2016Improved Techniques for Training GANscited by 2,938
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
38 more
  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  5. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  7. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  8. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  9. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  11. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  12. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  13. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  14. 2018Object Hallucination in Image Captioningcited by 661
  15. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  16. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  17. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  18. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  19. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  20. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  21. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  22. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  23. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  24. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  25. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  26. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  27. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  28. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  29. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  30. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  31. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  32. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 293
  33. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  34. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  35. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  36. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  37. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 255
  38. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 254

50 most cited of 168 All 168 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 19; 2026: 6.

27 papers.

Sort:

By area:

  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  2. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  3. 2025MLLMs are Deeply Affected by Modality Biascited by 38
  4. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  5. 2025VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Modelscited by 31
  6. 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Researchcited by 30
  7. 2025Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoningcited by 28
  8. 2025Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methodscited by 15
  9. 2025WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in WebUI-to-Codecited by 14
  10. 2025MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Modelscited by 13
  11. 2025Unified Multimodal Understanding via Byte-Pair Visual Encodingcited by 9
  12. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Modelscited by 7
15 more
  1. 2025FlySearch: Exploring how vision-language models explorecited by 7
  2. 2025Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Modelscited by 5
  3. 2025MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Modelscited by 5
  4. 2026AnimeAgent: Is the Multi-Agent via Image-to-Video models a Good Disney Storytelling Artist?cited by 5
  5. 2024Seeing is Deceiving: Exploitation of Visual Pathways in Multi-Modal Language Modelscited by 4
  6. 2025Zero-shot Compositional Action Recognition with Neural Logic Constraintscited by 4
  7. 2025Multimodal Representation Learning and Fusioncited by 3
  8. 2025Multi-modal Data Spectrum: Multi-modal Datasets are Multi-dimensionalcited by 3
  9. 2026Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMscited by 3
  10. 2026EquiMem: Calibrating Shared Memory in Multi-Agent Debate via Game-Theoretic Equilibriumcited by 3
  11. 2025A Systematic Review of Open Datasets Used in Text-to-Image (T2I) Gen AI Model Safetycited by 2
  12. 2026SinkTrack: Attention Sink based Context Anchoring for Large Language Modelscited by 1
  13. 2025Aligning MLLM Benchmark With Human Preferences via Structural Equation Modelingcited by 0
  14. 2026MaD-Mix: Multi-Modal Data Mixtures via Latent Space Coupling for Vision-Language Model Trainingcited by 0
  15. 2026Soft Spatial Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveys
  2. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria
  3. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  4. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning
  5. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  6. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
  7. 2025Benchmarking Large and Small MLLMs
  8. 2025On Path to Multimodal Generalist: General-Level and General-Bench