mlsanityData to 14 Oct 2026

VISTA: Auditing Semantic Divergence in Vision-Language Models

arXiv 2607.02995 ↗First posted 3 Jul 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models can exhibit visual concept-conditioned divergence: given images containing demographic features, corporate logos, or ideological symbols, some models produce unusually uniform responses that differ from what peer models say about the same input. These behaviors evade text-only audits because visual concepts cannot be isolated or substituted the way text tokens can. We present VISTA (Visual Inconsistency Screening Through Analysis), a black-box cross-model audit that couples semantic entropy with distribution-based divergence to flag model-specific anomalies. In a controlled study, we implant concept-conditioned stances in three VLMs via fine-tuning on small biased datasets and confirm that VISTA detects them. Auditing six VLMs across 19 topics, VISTA surfaces 142 high-suspicion cases (1.2%) and identifies selective refusal as a previously unreported divergence pattern, where models refuse demographic queries at rates varying from 0 to 65% across groups.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 31 references that are in the map. First posted on arXiv 3 Jul 2026.

Builds on

24 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024MM-R3: On (In-)Consistency of Vision-Language Models (VLMs)7 Oct
  2. 2024Unveiling the Tapestry of Consistency in Large Vision-Language Models23 May

Other references · 22

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  7. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  8. 2019STRIP: A Defence Against Trojan Attacks on Deep Neural Networkscited by 459
  9. 2019FairFace: Face Attribute Dataset for Balanced Race, Gender, and Agecited by 87
  10. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Modelscited by 87
  11. 2023CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learningcited by 63
  12. 2024Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" Cyclecited by 32
10 more
  1. 2024Uncertainty-Aware Evaluation for Vision-Language Modelscited by 29
  2. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuningcited by 28
  3. 2025Backdoor Attack on Vision Language Models with Stealthy Semantic Manipulationcited by 14
  4. 2023TIJO: Trigger Inversion with Joint Optimization for Defending Multimodal Backdoored Modelscited by 12
  5. 2025ViLU: Learning Vision-Language Uncertainties for Failure Predictioncited by 9
  6. 2025Bridging the Gap in Vision Language Models in Identifying Unsafe Concepts Across Modalitiescited by 4
  7. 2025Propaganda AI: An Analysis of Semantic Divergence in Large Language Modelscited by 3
  8. 2025Improving the Sensitivity of Backdoor Detectors via Class Subspace Orthogonalizationcited by 3
  9. 2025Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision-Language Modelscited by 2
  10. 2025Concept-Guided Backdoor Attack on Vision Language Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
  2. 2026Reliability Challenges in Diffusion Vision-Language Models
  3. 2024Uncovering Bias in Large Vision-Language Models with Counterfactuals
  4. 2026Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity?
  5. 2026Evian: Towards Explainable Visual Instruction-tuning Data Auditing
  6. 2026CDH-Bench: A Commonsense-Driven Hallucination Benchmark for Evaluating Visual Fidelity in Vision-Language Models
  7. 2026DistractMIA: Black-Box Membership Inference on Vision-Language Models via Semantic Distraction
  8. 2026V-FAT: Benchmarking Visual Fidelity Against Text-bias