mlsanityData to 7 Oct 2026

MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities

arXiv 2308.02490 ↗First posted 4 Aug 2023cs.AI cs.CL cs.CVcited by 1,221

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We propose MM-Vet, an evaluation benchmark that examines large multimodal models (LMMs) on complicated multimodal tasks. Recent LMMs have shown various intriguing abilities, such as solving math problems written on the blackboard, reasoning about events and celebrities in news images, and explaining visual jokes. Rapid model advancements pose challenges to evaluation benchmark development. Problems include: (1) How to systematically structure and evaluate the complicated multimodal tasks; (2) How to design evaluation metrics that work well across question and answer types; and (3) How to give model insights beyond a simple performance ranking. To this end, we present MM-Vet, designed based on the insight that the intriguing ability to solve complicated tasks is often achieved by a generalist model being able to integrate different core vision-language (VL) capabilities. MM-Vet defines 6 core VL capabilities and examines the 16 integrations of interest derived from the capability combination. For evaluation metrics, we propose an LLM-based evaluator for open-ended outputs. The evaluator enables the evaluation across different question types and answer styles, resulting in a unified scoring metric. We evaluate representative LMMs on MM-Vet, providing insights into the capabilities of different LMM system paradigms and models.

Key facts. Cited by 1,221 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 74 of its 93 references that are in the map. First posted on arXiv 4 Aug 2023.

Builds on

74 of 93 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  4. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  6. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 68

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  13. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  14. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  15. 2019Towards VQA Models That Can Readcited by 1,696
  16. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  17. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  18. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  19. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  20. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  21. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  22. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  23. 2023PaLM 2 Technical Reportcited by 1,154
  24. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  25. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  26. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  27. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  28. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  29. 2023Instruction Tuning with GPT-4cited by 676
  30. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  31. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  32. 2022PAL: Program-aided Language Modelscited by 635
  33. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  34. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  35. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  36. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  37. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  38. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 63; 2024: 404; 2025: 473; 2026: 281.

1,221 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  8. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  9. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  10. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  11. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  12. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
38 more
  1. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  2. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  3. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  4. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  6. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  7. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  8. 2025Qwen2.5-Omni Technical Reportcited by 704
  9. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  10. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  11. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  12. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  13. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  14. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  15. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  16. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  17. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  18. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  19. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  20. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  21. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  22. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  23. 2025Kimi-VL Technical Reportcited by 381
  24. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  25. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  26. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  27. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  28. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  29. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  30. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  31. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  32. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  33. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  34. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  35. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  36. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  37. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  38. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285

50 most cited of 1,221 All 1,221 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation
  2. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
  3. 2024MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Tasks
  4. 2025MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique
  5. 2024A Survey on Benchmarks of Multimodal Large Language Models
  6. 2024Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens
  7. 2026Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory
  8. 2025Evaluating Multimodal Large Language Models on Educational Textbook Question Answering