mlsanityData to 14 Oct 2026

Pathological Truth Bias in Vision-Language Models

arXiv 2509.22674 ↗First posted 14 Sep 2025cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision Language Models (VLMs) are improving quickly, but standard benchmarks can hide systematic failures that reduce real world trust. We introduce MATS (Multimodal Audit for Truthful Spatialization), a compact behavioral audit that measures whether models reject visually contradicted statements, and two metrics Spatial Consistency Score (SCS) and Incorrect Agreement Rate (IAR). Instruction tuned generative VLMs (LLaVA 1.5, QwenVLchat) exhibit very low SCS and high IAR, while contrastive encoders (CLIP, SigLIP) are far more robust. Activation patching causally localizes failure loci (mid to late cross attention for generative models, pooled projection components for contrastive models) and suggests concrete repair paths.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 39 references that are in the map. First posted on arXiv 14 Sep 2025.

Builds on

36 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  10. 2020Learning to summarize from human feedbackcited by 2,588
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  12. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
23 more
  1. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  3. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  4. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  5. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  6. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  7. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  8. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  9. 2018Object Hallucination in Image Captioningcited by 669
  10. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  11. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  12. 2021Causal Abstractions of Neural Networkscited by 422
  13. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 316
  14. 2022Visual Spatial Reasoningcited by 293
  15. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  16. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  17. 2024How to use and interpret activation patchingcited by 214
  18. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 208
  19. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  20. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Modelscited by 113
  21. 2023Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Biascited by 53
  22. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resamplingcited by 33
  23. 2025Mitigating Image Captioning Hallucinations in Vision-Language Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Mechanism-Level Evaluation for Vision-Language Models: Controlled Activation-Replacement Diagnosis of Gender Biascited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models
  2. 2025Test-Time Consistency in Vision Language Models
  3. 2026VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models
  4. 2026CDH-Bench: A Commonsense-Driven Hallucination Benchmark for Evaluating Visual Fidelity in Vision-Language Models
  5. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  6. 2025Synthetic Stimuli, Real Gains: Rethinking VLM Fine-Tuning Through Fully Controlled Data Generation
  7. 2026Semantic-Spatial Agreement Verification for Mitigating Object Hallucination in Multimodal Large Language Models
  8. 2026Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption