mlsanityData to 7 Oct 2026

MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

arXiv 2306.13394 ↗First posted 23 Jun 2023cs.CVcited by 1,624

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Model (MLLM) relies on the powerful LLM to perform multimodal tasks, showing amazing emergent abilities in recent studies, such as writing poems based on an image. However, it is difficult for these case studies to fully reflect the performance of MLLM, lacking a comprehensive evaluation. In this paper, we fill in this blank, presenting the first comprehensive MLLM Evaluation benchmark MME. It measures both perception and cognition abilities on a total of 14 subtasks. In order to avoid data leakage that may arise from direct use of public datasets for evaluation, the annotations of instruction-answer pairs are all manually designed. The concise instruction design allows us to fairly compare MLLMs, instead of struggling in prompt engineering. Besides, with such an instruction, we can also easily carry out quantitative statistics. A total of 30 advanced MLLMs are comprehensively evaluated on our MME, which not only suggests that existing MLLMs still have a large room for improvement, but also reveals the potential directions for the subsequent model optimization. The data are released at the project page https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Evaluation.

Key facts. Cited by 1,624 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 72 references that are in the map. First posted on arXiv 23 Jun 2023.

Builds on

41 of 72 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  3. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May
  4. 2023VPGTrans: Transfer Visual Prompt Generator across LLMs2 May

Other references · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2023A Survey of Large Language Modelscited by 2,537
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
25 more
  1. 2015VQA: Visual Question Answeringcited by 2,489
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  6. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  7. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  10. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  11. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  12. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  13. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  14. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  15. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  16. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  17. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  18. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  19. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  20. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  21. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  22. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  23. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 100
  24. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 64
  25. 2020Geometry-guided Dense Perspective Network for Speech-Driven Facial Animationcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 101; 2024: 480; 2025: 584; 2026: 459.

1,624 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  8. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  9. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  10. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  11. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
38 more
  1. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  2. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  5. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  7. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  8. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  9. 2025Qwen2.5-Omni Technical Reportcited by 704
  10. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  11. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  12. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  13. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  14. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  15. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  16. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  17. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  18. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  19. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  20. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  21. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  22. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  23. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  24. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  25. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  26. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  27. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  28. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  29. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  30. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  31. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  32. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  33. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  34. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  35. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  36. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  37. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  38. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297

50 most cited of 1,624 All 1,624 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026What We are Missing in Multimodal LLM Evaluation?
  2. 2024A Survey on Benchmarks of Multimodal Large Language Models
  3. 2025MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models
  4. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
  5. 2025Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark
  6. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveys
  7. 2024Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens
  8. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks