mlsanityData to 7 Oct 2026

MMBench: Is Your Multi-modal Model an All-around Player?

arXiv 2307.06281 ↗First posted 12 Jul 2023cs.CV cs.CLcited by 2,105

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Large vision-language models (VLMs) have recently achieved remarkable progress, exhibiting impressive multimodal perception and reasoning abilities. However, effectively evaluating these large VLMs remains a major challenge, hindering future development in this domain. Traditional benchmarks like VQAv2 or COCO Caption provide quantitative performance measurements but lack fine-grained ability assessment and robust evaluation metrics. Meanwhile, subjective benchmarks, such as OwlEval, offer comprehensive evaluations of a model's abilities by incorporating human labor, which is not scalable and may display significant bias. In response to these challenges, we propose MMBench, a bilingual benchmark for assessing the multi-modal capabilities of VLMs. MMBench methodically develops a comprehensive evaluation pipeline, primarily comprised of the following key features: 1. MMBench is meticulously curated with well-designed quality control schemes, surpassing existing similar benchmarks in terms of the number and variety of evaluation questions and abilities; 2. MMBench introduces a rigorous CircularEval strategy and incorporates large language models to convert free-form predictions into pre-defined choices, which helps to yield accurate evaluation results for models with limited instruction-following capabilities. 3. MMBench incorporates multiple-choice questions in both English and Chinese versions, enabling an apples-to-apples comparison of VLMs' performance under a bilingual context. To summarize, MMBench is a systematically designed objective benchmark for a robust and holistic evaluation of vision-language models. We hope MMBench will assist the research community in better evaluating their models and facilitate future progress in this area. The evalutation code of MMBench has been integrated into VLMEvalKit: https://github.com/open-compass/VLMEvalKit.

Key facts. Cited by 2,105 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 72 references that are in the map. First posted on arXiv 12 Jul 2023.

Builds on

40 of 72 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model28 Apr
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep

Other references · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
26 more
  1. 2019Towards VQA Models That Can Readcited by 1,696
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  9. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  11. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  13. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  14. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  15. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  16. 2018nocaps: novel object captioning at scalecited by 444
  17. 2019KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessmentcited by 351
  18. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  19. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  20. 2022Visual Spatial Reasoningcited by 288
  21. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  22. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  23. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  24. 2017ShapeWorld - A new test methodology for multimodal language understandingcited by 37
  25. 2017Dual-Glance Model for Deciphering Social Relationshipscited by 21
  26. 2022Mitigating Representation Bias in Action Recognition: Algorithms and Benchmarkscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 89; 2024: 578; 2025: 790; 2026: 648.

2,105 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  6. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  8. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  9. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  10. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  12. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
38 more
  1. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  3. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  4. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  5. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  6. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  8. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  9. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  10. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  11. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  12. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  13. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  14. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  15. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  16. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  17. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  18. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  19. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  20. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  21. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  22. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  23. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 489
  24. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  25. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  26. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  27. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  28. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  29. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  30. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  31. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  32. 2025Kimi-VL Technical Reportcited by 381
  33. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  34. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  35. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  36. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  37. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  38. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344

50 most cited of 2,105 All 2,105 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks
  2. 2024A Survey on Benchmarks of Multimodal Large Language Models
  3. 2024MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Tasks
  4. 2026Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models
  5. 2025Judge Anything: MLLM as a Judge Across Any Modality
  6. 2024GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation
  7. 2026JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation
  8. 2025IQBench: How "Smart'' Are Vision-Language Models? A Study with Human IQ Tests