mlsanityData to 7 Oct 2026

When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias

arXiv 2604.17768 ↗First posted 20 Apr 2026cs.AIcited by 3

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The reliability of VLM-as-a-Judge is critical for the automatic evaluation of vision-language models (VLMs). Despite recent progress, our analysis reveals that VLM-as-a-Judge often pays limited attention to the image when making decisions. Instead, they often blindly favor the more informative answer, even when they can recognize it conflicts with the image content. We call this problem informativeness bias, which significantly undermines judge reliability. To address it, we propose BIRCH (Balanced Informativeness and CoRrectness with a Truthful AnCHor), a judging paradigm that first corrects inconsistencies with the image content in candidate answers, and then compares the answers against this corrected version. This shifts the judge's focus from informativeness to image-grounded correctness. Experiments on multiple models and benchmarks show that BIRCH reduces informativeness bias by up to 17%, resulting in performance gains of up to 9.8%. Our work reveals an overlooked but fundamental flaw in current VLM-as-a-Judge systems and highlights the need for more principled designs.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 44 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

29 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples18 Oct

Other references · 28

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2024A Survey on LLM-as-a-Judgecited by 1,384
  9. 2023LIMA: Less Is More for Alignmentcited by 1,080
  10. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  11. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  12. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
16 more
  1. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  2. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  3. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  4. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  5. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  6. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  7. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  8. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  9. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
  10. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  11. 2024Explaining Length Bias in LLM-Based Preference Evaluationscited by 48
  12. 2025MLLMs are Deeply Affected by Modality Biascited by 38
  13. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 33
  14. 2024RevisEval: Improving LLM-as-a-Judge via Response-Adapted Referencescited by 19
  15. 2023Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying Referencescited by 11
  16. 2025Graders should cheat: privileged information enables expert-level automated evaluationscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Don't Read the Log: Execution Traces Contaminate Verifiers in Video-Generation Agentscited by 0
  2. 2026A Low Grounding Score Is Not an Ungrounded Judge: Identifying the Perceptibility Confound in Multimodal Oversightcited by 0
  3. 2026Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
  2. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
  3. 2025Vision Language Models are Biased
  4. 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models
  5. 2026MMErroR: A Benchmark for Erroneous Reasoning in Vision-Language Models
  6. 2025Believing without Seeing: Quality Scores for Contextualizing Vision-Language Model Explanations
  7. 2026MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
  8. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs