mlsanityData to 7 Oct 2026

MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs

arXiv 2411.15296 ↗First posted 22 Nov 2024cs.CV cs.AI cs.CLcited by 72

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

As a prominent direction of Artificial General Intelligence (AGI), Multimodal Large Language Models (MLLMs) have garnered increased attention from both industry and academia. Building upon pre-trained LLMs, this family of models further develops multimodal perception and reasoning capabilities that are impressive, such as writing code given a flow chart or creating stories based on an image. In the development process, evaluation is critical since it provides intuitive feedback and guidance on improving models. Distinct from the traditional train-eval-test paradigm that only favors a single task like image classification, the versatility of MLLMs has spurred the rise of various new benchmarks and evaluation methods. In this paper, we aim to present a comprehensive survey of MLLM evaluation, discussing four key aspects: 1) the summarised benchmarks types divided by the evaluation capabilities, including foundation capabilities, model self-analysis, and extented applications; 2) the typical process of benchmark counstruction, consisting of data collection, annotation, and precautions; 3) the systematic evaluation manner composed of judge, metric, and toolkit; 4) the outlook for the next benchmark. This work aims to offer researchers an easy grasp of how to effectively evaluate MLLMs according to different needs and to inspire better evaluation methods, thereby driving the progress of MLLM research.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 161 of its 231 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

161 of 231 references.

Closest ancestors · 34

Same direction, newest first

  1. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?23 Aug
  2. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLM9 Aug
  3. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models17 Jul
  4. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs1 Jul
  5. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation29 Jun
  6. 2024MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs24 Jun
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  8. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs17 Jun
  9. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding13 Jun
  10. 2024TRINS: Towards Multimodal Language Models that Can Read10 Jun
  11. 2024Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models3 May
  12. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension25 Apr
22 more
  1. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI24 Apr
  2. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  3. 2024ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Capability for Large Vision-Language Models29 Mar
  4. 2024CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning25 Jan
  5. 2024CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark22 Jan
  6. 2024Benchmarking Large Multimodal Models against Common Corruptions22 Jan
  7. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences19 Jan
  8. 2023Silkie: Preference Distillation for Large Visual Language Models17 Dec
  9. 2023BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models5 Dec
  10. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models28 Nov
  11. 2023InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models20 Nov
  12. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents9 Nov
  13. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  14. 2023TouchStone: Evaluating Vision-Language Models by Language Models31 Aug
  15. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use12 Aug
  16. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  17. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  18. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  19. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  20. 2023A Survey on Multimodal Large Language Models23 Jun
  21. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  22. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun

Other references · 127

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  10. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
38 more
  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  3. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  4. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  7. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  8. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  9. 2023Instruction Tuning with GPT-4cited by 676
  10. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  11. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  12. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  13. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  14. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  15. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  16. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  17. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  18. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  19. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  20. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  21. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  22. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  23. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  24. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  25. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  26. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  27. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  28. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  29. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  30. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  31. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  32. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 293
  33. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  34. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  35. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  36. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  37. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  38. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245

50 most cited of 127 All 127 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 49; 2026: 21.

72 papers.

Sort:

By area:

  1. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  2. 2025Thyme: Think Beyond Imagescited by 160
  3. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  4. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
  5. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  6. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  7. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  8. 2025MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversationcited by 28
  9. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  10. 2025FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluationcited by 27
  11. 2025MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Modelscited by 24
  12. 2025Robix: A Unified Model for Robot Interaction, Reasoning and Planningcited by 23
38 more
  1. 2025STAR-R1: Spatial TrAnsformation Reasoning by Reinforcing Multimodal LLMscited by 21
  2. 2025HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluationcited by 19
  3. 2025Seeing Before Reasoning: A Unified Framework for Generalizable and Explainable Fake Image Detectioncited by 19
  4. 2025MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarioscited by 18
  5. 2025MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Modelscited by 17
  6. 2025SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inferencecited by 15
  7. 2025Aligning Multimodal LLM with Human Preference: A Surveycited by 12
  8. 2025Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learningcited by 11
  9. 2025Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Modelscited by 11
  10. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
  11. 2025MCITlib: Multimodal Continual Instruction Tuning Library and Benchmarkcited by 10
  12. 2025Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMscited by 9
  13. 2024INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuningcited by 8
  14. 2025PhysicsArena: The First Multimodal Physics Reasoning Benchmark Exploring Variable, Process, and Solution Dimensionscited by 8
  15. 2025VideoAds for Fast-Paced Video Understandingcited by 7
  16. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Modelscited by 7
  17. 2025Evidence-Grounded Trustworthy Multimodal Reasoning and Evaluation Benchmark in Complex Urban Scenescited by 7
  18. 2025VFaith: Do Large Multimodal Models Really Reason on Seen Images Rather than Previous Memories?cited by 7
  19. 2025When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorationscited by 6
  20. 2025Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculationcited by 6
  21. 2025KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large Vision-Language Models in the Korean Languagecited by 5
  22. 2025From Correctness to Comprehension: AI Agents for Personalized Error Diagnosis in Educationcited by 4
  23. 2025RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answeringcited by 4
  24. 2025"See the World, Discover Knowledge": A Chinese Factuality Evaluation for Large Vision Language Modelscited by 3
  25. 2025Reimagining Urban Science: Scaling Causal Inference with Large Language Modelscited by 3
  26. 2025Abstractive Visual Understanding of Multi-modal Structured Knowledge: A New Perspective for MLLM Evaluationcited by 3
  27. 2025Image Corruption-Inspired Membership Inference Attacks against Large Vision-Language Modelscited by 3
  28. 2025SMA: Who Said That? Auditing Membership Leakage in Semi-Black-box RAG Controllingcited by 3
  29. 2026SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understandingcited by 3
  30. 2026VisPhyWorld: Probing Physical Reasoning via Code-Driven Video Reconstructioncited by 3
  31. 2026QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compressioncited by 3
  32. 2026Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbationcited by 3
  33. 2026Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Docited by 3
  34. 2025E3RG: Building Explicit Emotion-driven Empathetic Response Generation System with Multimodal Large Language Modelcited by 2
  35. 2026TPRU: Advancing Temporal and Procedural Understanding in Large Multimodal Modelscited by 2
  36. 2026TokenCom: Vision-Language Model for Multimodal and Multitask Token Communicationscited by 2
  37. 2025BYO-Eval: Build Your Own Dataset for Fine-Grained Visual Assessment of Multimodal Language Modelscited by 1
  38. 2025COREVQA: A Crowd Observation and Reasoning Entailment Visual Question Answering Benchmarkcited by 1

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria
  2. 2026What We are Missing in Multimodal LLM Evaluation?
  3. 2024A Survey on Evaluation of Multimodal Large Language Models
  4. 2024A Survey on Benchmarks of Multimodal Large Language Models
  5. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
  6. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  7. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveys
  8. 2024Understanding the Role of LLMs in Multimodal Evaluation Benchmarks