mlsanityData to 7 Oct 2026

MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities

arXiv 2408.00765 ↗First posted 1 Aug 2024cs.CV cs.AI cs.CLcited by 57

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

MM-Vet, with open-ended vision-language questions targeting at evaluating integrated capabilities, has become one of the most popular benchmarks for large multimodal model evaluation. MM-Vet assesses six core vision-language (VL) capabilities: recognition, knowledge, spatial awareness, language generation, OCR, and math. However, its question format is restricted to single image-text pairs, lacking the interleaved image and text sequences prevalent in real-world scenarios. To address this limitation, we introduce MM-Vet v2, which includes a new VL capability called "image-text sequence understanding", evaluating models' ability to process VL sequences. Furthermore, we maintain the high quality of evaluation samples while further expanding the evaluation set size. Using MM-Vet v2 to benchmark large multimodal models, we found that Claude 3.5 Sonnet is the best model with a score of 71.8, slightly outperforming GPT-4o which scored 71.0. Among open-weight models, InternVL2-Llama3-76B leads with a score of 68.4. The code, data, and leaderboard are accessible at https://github.com/yuweihao/MM-Vet.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 29 references that are in the map. First posted on arXiv 1 Aug 2024.

Builds on

23 of 29 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model29 Jan
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  3. 2023Generative Multimodal Models are In-Context Learners20 Dec
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  5. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  9. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 14

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  7. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  8. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  9. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  10. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
  11. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  12. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
2 more
  1. 2023MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learningcited by 46
  2. 2024VideoGUI: A Benchmark for GUI Automation from Instructional Videoscited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 35; 2026: 16.

57 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2025Thyme: Think Beyond Imagescited by 160
  4. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  5. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  6. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 54
  7. 2025Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuningcited by 31
  8. 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Modelscited by 29
  9. 2025SITE: towards Spatial Intelligence Thorough Evaluationcited by 26
  10. 2025ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMscited by 25
  11. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  12. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?cited by 19
38 more
  1. 2025Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysiscited by 16
  2. 2024Phantom of Latent for Large Language and Vision Modelscited by 15
  3. 2025Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methodscited by 15
  4. 2025Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoningcited by 14
  5. 2025Generative RLHF-V: Learning Principles from Multi-modal Human Preferencecited by 13
  6. 2025Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learningcited by 11
  7. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  8. 2024VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Modelscited by 10
  9. 2025V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Modelscited by 10
  10. 2025MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Modelscited by 10
  11. 2025OralGPT-Omni: A Versatile Dental Multimodal Large Language Modelcited by 10
  12. 2026Vero: An Open RL Recipe for General Visual Reasoningcited by 10
  13. 2025GenRecal: Generation after Recalibration from Large to Small Vision-Language Modelscited by 9
  14. 2026Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVRcited by 9
  15. 2025Unified Reinforcement and Imitation Learning for Vision-Language Modelscited by 8
  16. 2025OMNIGUARD: An Efficient Approach for AI Safety Moderation Across Languages and Modalitiescited by 5
  17. 2025VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agentscited by 5
  18. 2025RGB-Th-Bench: A Dense benchmark for Visual-Thermal Understanding of Vision Language Modelscited by 4
  19. 2025Draw-In-Mind: Rebalancing Designer-Painter Roles in Unified Multimodal Models Benefits Image Editingcited by 4
  20. 2025VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenescited by 4
  21. 2025LEC: Linear Expectation Constraints for Selection-Conditioned Risk Control in Selective Prediction and Routing Systemscited by 4
  22. 2025Reassessing the Role of Supervised Fine-Tuning: An Empirical Study in VLM Reasoningcited by 4
  23. 2026OralGPT-Plus: Learning to Use Visual Tools via Reinforcement Learning for Panoramic X-ray Analysiscited by 4
  24. 2026Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encodingcited by 4
  25. 2025FlagEvalMM: A Flexible Framework for Comprehensive Multimodal Model Evaluationcited by 3
  26. 2025MOAT: Evaluating LMMs for Capability Integration and Instruction Groundingcited by 2
  27. 2025PRISM: Robust VLM Alignment with Principled Reasoning for Integrated Safety in Multimodalitycited by 2
  28. 2025Distilling Counterfactual Reasoning from Language to Vision: Causal Graph Guided Post-Training for Video Understandingcited by 2
  29. 2026MetaphorStar: Image Metaphor Understanding and Reasoning with End-to-End Visual Reinforcement Learningcited by 2
  30. 2025Learning to Detect Unseen Jailbreak Attacks in Large Vision-Language Modelscited by 1
  31. 2025Learning to Detect Unknown Jailbreak Attacks in Large Vision-Language Modelscited by 1
  32. 2025Masked Diffusion Captioning for Visual Feature Learningcited by 1
  33. 2026MM-THEBench: Do Reasoning MLLMs Think Reasonably?cited by 1
  34. 2026BreastGPT: A Multimodal Large Language Model for the Full Spectrum of Breast Cancer Clinical Routinecited by 1
  35. 2026Knowing the Self, Understanding the World: A Dual-Cognition Benchmark for UAV Spatio-temporal Reasoning with MLLMscited by 1
  36. 2025Towards Interpretable Renal Health Decline Forecasting via Multi-LMM Collaborative Reasoning Frameworkcited by 0
  37. 2025When Harmful Content Gets Camouflaged: Unveiling Perception Failure of LVLMs with CamHarmTIcited by 0
  38. 2026UniFinEval: Towards Unified Evaluation of Financial Multimodal Models across Text, Images and Videoscited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
  2. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models
  3. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
  4. 2024MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Tasks
  5. 2024MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models
  6. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
  7. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation
  8. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions