mlsanityData to 7 Oct 2026

MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria

arXiv 2311.13951 ↗First posted 23 Nov 2023cs.CLcited by 27

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal large language models (MLLMs) have broadened the scope of AI applications. Existing automatic evaluation methodologies for MLLMs are mainly limited in evaluating queries without considering user experiences, inadequately addressing the nuances of creative and associative multimodal tasks. However, the open-ended and subjective nature of such tasks poses a significant challenge to the evaluation methodology, where it is difficult to define the ground-truth answers for them. To this end, in our paper, we propose a new evaluation paradigm for MLLMs, which is evaluating MLLMs with per-sample criteria using potent MLLM as the judge. To validate the feasibility and effectiveness of this paradigm, we design a benchmark, dubbed MLLM-Bench, by curating the evaluation samples across six comprehensive cognitive levels. We benchmark 21 popular MLLMs in a pairwise-comparison fashion, showing diverse performance across models. Moreover, the validity of our benchmark manifests itself in reaching 88.02% agreement with human evaluation. We contend that the proposed paradigm explores the potential of MLLMs as effective evaluation tools with the help of per-sample criteria. See online leaderboard at urlhttps://mllm-bench.llmzoo.com.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 42 references that are in the map. First posted on arXiv 23 Nov 2023.

Builds on

23 of 42 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023TouchStone: Evaluating Vision-Language Models by Language Models31 Aug
  3. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use12 Aug
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  5. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  8. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun

Other references · 15

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  7. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  8. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  9. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  10. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  11. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  12. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 143
3 more
  1. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 118
  2. 2023Style Over Substance: Evaluation Biases for Large Language Modelscited by 68
  3. 2023Phoenix: Democratizing ChatGPT across Languagescited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 8; 2026: 8.

27 papers.

Sort:

By area:

  1. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  2. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 56
  3. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 54
  4. 2024VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMscited by 47
  5. 2025ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluationcited by 35
  6. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  7. 2024BlenderLLM: Training Large Language Models for Computer-Aided Design with Self-improvementcited by 26
  8. 2025ViLBench: A Suite for Vision-Language Process Reward Modelingcited by 16
  9. 2023On the Promises and Challenges of Multimodal Foundation Models for Geographical, Environmental, Agricultural, and Urban Planning Applicationscited by 15
  10. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  11. 2025Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLMcited by 14
  12. 2025ReLook: Vision-Grounded RL with a Multimodal LLM Critic for Agentic Web Codingcited by 12
15 more
  1. 2024Exploring Boundary of GPT-4V on Marine Analysis: A Preliminary Case Studycited by 9
  2. 2026MUSE: Benchmarking Manufacturable, Functional, and Assemblable Text-to-CAD Generationcited by 8
  3. 2024MULTI: Multimodal Understanding Leaderboard with Text and Imagescited by 7
  4. 2026WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Modelscited by 6
  5. 2024How Can Large Language Models Enable Better Socially Assistive Human-Robot Interaction: A Brief Surveycited by 4
  6. 2024Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMscited by 3
  7. 2026World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoningcited by 3
  8. 2025URECA: Unique Region Caption Anythingcited by 2
  9. 2025VCU-Bridge: Hierarchical Visual Connotation Understanding via Semantic Bridgingcited by 2
  10. 2026Prototypicality Bias Reveals Blindspots in Multimodal Evaluation Metricscited by 2
  11. 2026Bi-Level Prompt Optimization for Multimodal LLM-as-a-Judgecited by 1
  12. 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Modelscited by 1
  13. 2025Evaluating Cultural Knowledge Processing in Large Language Models: A Cognitive Benchmarking Framework Integrating Retrieval-Augmented Generationcited by 0
  14. 2026V-Rubrics: Visual Faithfulness via Rubric-Based Reinforcement Learningcited by 0
  15. 2026CollageAttack: Exploiting Cross-Modal Alignment Flaws in T2I Models through Spatial Text Compositioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
  2. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  3. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
  4. 2024A Survey on Benchmarks of Multimodal Large Language Models
  5. 2026What We are Missing in Multimodal LLM Evaluation?
  6. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveys
  7. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning
  8. 2024Understanding the Role of LLMs in Multimodal Evaluation Benchmarks