# MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities

arXiv: https://arxiv.org/abs/2408.00765 · HTML: https://mlsanity.com/paper/2408.00765/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 1 Aug 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 23 of 23)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [CogVLM: Visual Expert for Pretrained Language Models](https://mlsanity.com/paper/2311.03079/) (2023)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models](https://mlsanity.com/paper/2308.01390/) (2023)
- [Otter: A Multi-Modal Model with In-Context Instruction Tuning](https://mlsanity.com/paper/2305.03726/) (2023)
- [Generative Multimodal Models are In-Context Learners](https://mlsanity.com/paper/2312.13286/) (2023)
- [InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model](https://mlsanity.com/paper/2401.16420/) (2024)
- [Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs](https://mlsanity.com/paper/2404.05719/) (2024)
- [ChartLlama: A Multimodal LLM for Chart Understanding and Generation](https://mlsanity.com/paper/2311.16483/) (2023)
- [GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation](https://mlsanity.com/paper/2311.07562/) (2023)
- [MM-VID: Advancing Video Understanding with GPT-4V(ision)](https://mlsanity.com/paper/2310.19773/) (2023)
- [MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning](https://mlsanity.com/paper/2311.17435/) (2023)
- [VideoGUI: A Benchmark for GUI Automation from Instructional Videos](https://mlsanity.com/paper/2406.10227/) (2024)

## Built on by (most cited first, 30 of 57)

- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Thyme: Think Beyond Images](https://mlsanity.com/paper/2508.11630/) (2025)
- [SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement](https://mlsanity.com/paper/2504.07934/) (2025)
- [Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2505.02567/) (2025)
- [OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs](https://mlsanity.com/paper/2510.10689/) (2025)
- [Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuning](https://mlsanity.com/paper/2505.19702/) (2025)
- [MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models](https://mlsanity.com/paper/2410.09733/) (2024)
- [SITE: towards Spatial Intelligence Thorough Evaluation](https://mlsanity.com/paper/2505.05456/) (2025)
- [ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs](https://mlsanity.com/paper/2506.10128/) (2025)
- [Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)](https://mlsanity.com/paper/2504.03151/) (2025)
- [AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?](https://mlsanity.com/paper/2410.21259/) (2024)
- [Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysis](https://mlsanity.com/paper/2509.09254/) (2025)
- [Phantom of Latent for Large Language and Vision Models](https://mlsanity.com/paper/2409.14713/) (2024)
- [Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods](https://mlsanity.com/paper/2510.07143/) (2025)
- [Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning](https://mlsanity.com/paper/2512.00818/) (2025)
- [Generative RLHF-V: Learning Principles from Multi-modal Human Preference](https://mlsanity.com/paper/2505.18531/) (2025)
- [V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Models](https://mlsanity.com/paper/2504.06148/) (2025)
- [Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learning](https://mlsanity.com/paper/2506.07227/) (2025)
- [OralGPT-Omni: A Versatile Dental Multimodal Large Language Model](https://mlsanity.com/paper/2511.22055/) (2025)
- [Vero: An Open RL Recipe for General Visual Reasoning](https://mlsanity.com/paper/2604.04917/) (2026)
- [Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR](https://mlsanity.com/paper/2605.20164/) (2026)
- [Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspective](https://mlsanity.com/paper/2410.22217/) (2024)
- [VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models](https://mlsanity.com/paper/2412.01822/) (2024)
- [GenRecal: Generation after Recalibration from Large to Small Vision-Language Models](https://mlsanity.com/paper/2506.15681/) (2025)
- [MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Models](https://mlsanity.com/paper/2510.16641/) (2025)
- [Unified Reinforcement and Imitation Learning for Vision-Language Models](https://mlsanity.com/paper/2510.19307/) (2025)
- [OMNIGUARD: An Efficient Approach for AI Safety Moderation Across Languages and Modalities](https://mlsanity.com/paper/2505.23856/) (2025)
- [LEC: Linear Expectation Constraints for Selection-Conditioned Risk Control in Selective Prediction and Routing Systems](https://mlsanity.com/paper/2512.01556/) (2025)
- [Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encoding](https://mlsanity.com/paper/2606.10738/) (2026)

## Abstract (arXiv)

MM-Vet, with open-ended vision-language questions targeting at evaluating integrated capabilities, has become one of the most popular benchmarks for large multimodal model evaluation. MM-Vet assesses six core vision-language (VL) capabilities: recognition, knowledge, spatial awareness, language generation, OCR, and math. However, its question format is restricted to single image-text pairs, lacking the interleaved image and text sequences prevalent in real-world scenarios. To address this limitation, we introduce MM-Vet v2, which includes a new VL capability called "image-text sequence understanding", evaluating models' ability to process VL sequences. Furthermore, we maintain the high quality of evaluation samples while further expanding the evaluation set size. Using MM-Vet v2 to benchmark large multimodal models, we found that Claude 3.5 Sonnet is the best model with a score of 71.8, slightly outperforming GPT-4o which scored 71.0. Among open-weight models, InternVL2-Llama3-76B leads with a score of 68.4. The code, data, and leaderboard are accessible at https://github.com/yuweihao/MM-Vet.