mlsanityData to 14 Oct 2026

It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them

arXiv 2609.37863 ↗First posted 29 Sep 2026cs.CL cs.AI cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning also fits: Social bias and fairness in VLMs

Abstract (arXiv)

Vision-language models (VLMs) are increasingly used in place of human annotators, making it important that substitutability tests reflect the model rather than incidental evaluation conditions. We introduce MIST, the Misleading-Image Stress Test: 200 English sentences, each built around a phrase readable either figuratively or literally and shown with an aligned image depicting its reading, a misleading image depicting the opposite, or no image at all. The guidelines require the label to be decided from the sentence alone, so no image should change any answer. We expected each image to pull a judge's labels toward the sense it depicts, and neither kind did. Across thirteen VLM judges, an aligned image changed 20.5% of labels and a misleading one 19.4%, close for every judge and both above the 11.6% produced by deleting the ignore-the-image instruction with the image left in place. Yet only 37% of the labels that differ between the two images moved toward the sense shown, and agreement with our human annotators is unchanged whether the image is absent, aligned or misleading. The effect is smaller in the seven judges that pass the alt-test than in the six that never do, but present in all of them: what moves a judge is that an image is there, not which of the two it is, so a substitutability verdict describes a configuration as much as a model.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 31 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

24 of 31 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Kimi-VL Technical Report10 Apr
  2. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?4 Mar
  3. 2024Understanding Figurative Meaning through Explainable Visual Entailment2 May

Other references · 21

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2025Qwen3-VL Technical Reportcited by 2,793
  7. 2025Gemma 3 Technical Reportcited by 1,824
  8. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  9. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  10. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  11. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  12. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
9 more
  1. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  2. 2022FLUTE: Figurative Language Understanding through Textual Explanationscited by 79
  3. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 74
  4. 2023IRFL: Image Recognition of Figurative Languagecited by 23
  5. 2024Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performancecited by 20
  6. 2024Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Modelscited by 15
  7. 2024Rolling the DICE on Idiomaticity: How LLMs Fail to Grasp Contextcited by 11
  8. 2025SemEval-2025 Task 1: AdMIRe -- Advancing Multimodal Idiomaticity Representationcited by 3
  9. 2025Can We Hide Machines in the Crowd? Quantifying Equivalence in LLM-in-the-loop Annotation Taskscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reading Right, Answering Wrong: How Visual Configuration Changes Affect Evidence Use in VLMs
  2. 2026When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias
  3. 2026Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagery
  4. 2025Fooling the LVLM Judges: Visual Biases in LVLM-Based Evaluation
  5. 2026Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs
  6. 2026Visual Input and Its Framing Affect Attribute-based Descriptions Produced by Large Vision-Language Models
  7. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
  8. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark