mlsanityData to 7 Oct 2026

MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark

arXiv 2402.04788 ↗First posted 7 Feb 2024cs.CL cs.AI cs.CVcited by 362

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have gained significant attention recently, showing remarkable potential in artificial general intelligence. However, assessing the utility of MLLMs presents considerable challenges, primarily due to the absence of multimodal benchmarks that align with human preferences. Drawing inspiration from the concept of LLM-as-a-Judge within LLMs, this paper introduces a novel benchmark, termed MLLM-as-a-Judge, to assess the ability of MLLMs in assisting judges across diverse modalities, encompassing three distinct tasks: Scoring Evaluation, Pair Comparison, and Batch Ranking. Our study reveals that, while MLLMs demonstrate remarkable human-like discernment in Pair Comparison, there is a significant divergence from human preferences in Scoring Evaluation and Batch Ranking. Furthermore, a closer examination reveals persistent challenges in the judgment capacities of LLMs, including diverse biases, hallucinatory responses, and inconsistencies in judgment, even in advanced models such as GPT-4V. These findings emphasize the pressing need for enhancements and further research efforts to be undertaken before regarding MLLMs as fully reliable evaluators. In light of this, we advocate for additional efforts dedicated to supporting the continuous development within the domain of MLLM functioning as judges. The code and dataset are publicly available at our project homepage: urlhttps://mllm-judge.github.io/.

Key facts. Cited by 362 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 74 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

50 of 74 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  2. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  3. 2023TouchStone: Evaluating Vision-Language Models by Language Models31 Aug
  4. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use12 Aug
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug

Other references · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  12. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
33 more
  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  3. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  4. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  5. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  6. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  7. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  8. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  9. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 376
  10. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 364
  11. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  12. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  13. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
  14. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 254
  15. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  16. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  17. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  18. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  19. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  20. 2023Generative Judge for Evaluating Alignmentcited by 163
  21. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  22. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Modelscited by 99
  23. 2023Verbosity Bias in Preference Labeling by Large Language Modelscited by 89
  24. 2023TrustGPT: A Benchmark for Trustworthy and Responsible Large Language Modelscited by 56
  25. 2024LLM-as-a-Coauthor: Can Mixed Human-Written and Machine-Generated Text Be Detected?cited by 47
  26. 2023Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluationcited by 46
  27. 2023Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4Vcited by 38
  28. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  29. 2023An Early Evaluation of GPT-4V(ision)cited by 31
  30. 2024I Think, Therefore I am: Benchmarking Awareness of Large Language Models Using AwareBenchcited by 20
  31. 2023A Closer Look into Automatic Evaluation Using Large Language Modelscited by 17
  32. 2023Low-code LLM: Graphical User Interface over Large Language Modelscited by 13
  33. 2023Aggregate, Decompose, and Fine-Tune: A Simple Yet Effective Factor-Tuning Method for Vision Transformercited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 44; 2025: 165; 2026: 153.

362 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  4. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  5. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  6. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  7. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  8. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  9. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  10. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  11. 2025AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectoriescited by 99
  12. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
38 more
  1. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  2. 2025PPTAgent: Generating and Evaluating Presentations Beyond Text-to-Slidescited by 83
  3. 2025Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedbackcited by 78
  4. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  5. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 72
  6. 2025KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Modelscited by 69
  7. 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 66
  8. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  9. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  10. 2024VLSBench: Unveiling Visual Leakage in Multimodal Safetycited by 59
  11. 2025MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Researchcited by 57
  12. 2025Video-Bench: Human-Aligned Video Generation Benchmarkcited by 53
  13. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 51
  14. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 45
  15. 2024MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPscited by 45
  16. 2024WebCode2M: A Real-World Dataset for Code Generation from Webpage Designscited by 43
  17. 2025DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generationcited by 41
  18. 2025UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learningcited by 41
  19. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Modelscited by 40
  20. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Eracited by 40
  21. 2025Optimizing Model Selection for Compound AI Systemscited by 34
  22. 2025Backdoor Cleaning without External Guidance in MLLM Fine-tuningcited by 34
  23. 2024MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMscited by 33
  24. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 33
  25. 2025AIGI-Holmes: Towards Explainable and Generalizable AI-Generated Image Detection via Multimodal Large Language Modelscited by 33
  26. 2025MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluationcited by 32
  27. 2024Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidancecited by 30
  28. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generationcited by 30
  29. 2024Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluationcited by 29
  30. 2024Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generationcited by 28
  31. 2025MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversationcited by 28
  32. 2025BlenderGym: Benchmarking Foundational Model Systems for Graphics Editingcited by 27
  33. 2025World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planningcited by 26
  34. 2025OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agentscited by 25
  35. 2025Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreementcited by 24
  36. 2025EditThinker: Unlocking Iterative Reasoning for Any Image Editorcited by 23
  37. 2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 23
  38. 2025nvAgent: Automated Data Visualization from Natural Language via Collaborative Agent Workflowcited by 21

50 most cited of 362 All 362 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria
  2. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
  3. 2026Lost in Translation: Do LVLM Judges Generalize Across Languages?
  4. 2024A Survey on Evaluation of Multimodal Large Language Models
  5. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning
  6. 2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments
  7. 2025On Path to Multimodal Generalist: General-Level and General-Bench
  8. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models