mlsanityData to 7 Oct 2026

MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI

arXiv 2404.16006 ↗First posted 24 Apr 2024cs.CVcited by 192

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large Vision-Language Models (LVLMs) show significant strides in general-purpose multimodal applications such as visual dialogue and embodied navigation. However, existing multimodal evaluation benchmarks cover a limited number of multimodal tasks testing rudimentary capabilities, falling short in tracking LVLM development. In this study, we present MMT-Bench, a comprehensive benchmark designed to assess LVLMs across massive multimodal tasks requiring expert knowledge and deliberate visual recognition, localization, reasoning, and planning. MMT-Bench comprises 31,325 meticulously curated multi-choice visual questions from various multimodal scenarios such as vehicle driving and embodied navigation, covering 32 core meta-tasks and 162 subtasks in multimodal understanding. Due to its extensive task coverage, MMT-Bench enables the evaluation of LVLMs using a task map, facilitating the discovery of in- and out-of-domain tasks. Evaluation results involving 30 LVLMs such as the proprietary GPT-4V, GeminiProVision, and open-sourced InternVL-Chat, underscore the significant challenges posed by MMT-Bench. We anticipate that MMT-Bench will inspire the community to develop next-generation multimodal foundation models aimed at achieving general-purpose multimodal intelligence.

Key facts. Cited by 192 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 62 references that are in the map. First posted on arXiv 24 Apr 2024.

Builds on

26 of 62 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model29 Jan
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  3. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  4. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  5. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  6. 2023TinyLVLM-eHub: Towards Comprehensive and Efficient Evaluation for Large Vision-Language Models7 Aug
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  8. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  11. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun

Other references · 15

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  9. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  11. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  12. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
3 more
  1. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  2. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  3. 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 94

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 53; 2025: 101; 2026: 38.

192 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  3. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  4. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  6. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  7. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  8. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  9. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  10. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  11. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
  12. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
38 more
  1. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
  2. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  3. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 123
  4. 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 114
  5. 2025Ovis2.5 Technical Reportcited by 103
  6. 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AIcited by 96
  7. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 89
  8. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  9. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  10. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 68
  11. 2025Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodologycited by 67
  12. 2025Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokenscited by 65
  13. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 64
  14. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 61
  15. 2024Needle In A Multimodal Haystackcited by 56
  16. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Modelscited by 53
  17. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 52
  18. 2024MuMA-ToM: Multi-modal Multi-Agent Theory of Mindcited by 51
  19. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
  20. 2025XLRS-Bench: Could Your Multimodal LLMs Understand Extremely Large Ultra-High-Resolution Remote Sensing Imagery?cited by 50
  21. 2024VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMscited by 47
  22. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  23. 2024VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Modelscited by 42
  24. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Modelscited by 41
  25. 2025MM-IFEngine: Towards Multimodal Instruction Followingcited by 41
  26. 2024Core Knowledge Deficits in Multi-Modal Language Modelscited by 38
  27. 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarioscited by 37
  28. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  29. 2024PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Modelscited by 36
  30. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  31. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  32. 2025EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Modelscited by 32
  33. 2025MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contextscited by 32
  34. 2025VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Modelscited by 31
  35. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generationcited by 30
  36. 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Researchcited by 30
  37. 2025SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectoriescited by 29
  38. 2026STEP3-VL-10B Technical Reportcited by 28

50 most cited of 192 All 192 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chaining
  2. 2025VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agents
  3. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  4. 2026MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models
  5. 2025RCI: A Score for Evaluating Global and Local Reasoning in Multimodal Benchmarks
  6. 2024JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images
  7. 2026FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs
  8. 2024MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities